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ABSTRACT 

An analysis of the incomprehensibility of large, complex computer systems is made. 
The thesis is that there Is strong relationship between system incomprehensibility 
and the necessity to trust computer systems. A cogent definition of Incomprehensi- 
bility In computer systems Is established, with common themes drawn from interdis- 
ciplinary literature dealing with computers and society. Reasons for the creation of 
incomprehensible computer systems are explored, as well as the consequences 
(both technical and social) of using and relying on them. The relationship between 
the real and perceived purposes of computer systems and the appropriateness of 
trusting these systems Is analyzed. Approaches for dealing with the existence of 
vital computer systems which are functionally incomprehensible are evaluated, and 
positive suggestions are made. 
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Chapter 1» Introduction 

The technological society contains many parts and specialized 
activities within a myriad of interconnections. The totality of 
such interconnections - the relationships of the parts to each 
other and the ©arts to the whole - Is something which is no 
longer comprehensible to anyone. In the complexity of this 
world, people are confronted wM* extraordinary events and 
functions that are literally unintelligible to them. They are 
unable to give an adequate explanation of man made 
phenomena in their immediate experience. They are unable to 
form a coherwrt, rational picture o| the whole, Uajar these cir- 
cumstances, all persons do and, indeed, must accept a great 
number of things on faith. They are awa/« that the major com- 
ponents of complex systems usually work, that other specialists 
know what tr»y«a doteg r arid 1^ 

together in relatively good adjustment. Their way of under- 
standing, however, is basicsHy religious rather than scientific; 
only a small portion of one's everyday experience In the tech- 
nological society can be tmim^vim^fk^f^^f^'^^f^ 9 
is forced to depend upon and have faith in matters about which 
one has Jrttte Information or Intelligent grasp- i*#«tWf ^condition 
that Ellul describes as the source of the modem versions of 

mystery, magic, and tea sacred. 
j ,, „—«— Langdon Winner 

In recent years, the computer science community has begun to recognize 
comprehensibility as an important dimension of computer programs. Much earlier, 
thoughtful observers of the growing preeminence of technology In modern cultures 
worried about the diminishing ability of people to understand and cope with the 
technological system with which they were so deeply Involved. Today, Dr. Robert 
Johnson, Vice-President for Engineering of Burroughs Corporation, is not alone in his 
belief that the most serious problem facing the computer Industry Is the 
incomprehensibility of large computer systems. 2 Several factors combine to make 
computer system incomprehensibility an issue of immediate concern to any society 



1 Langdon Winner, Autonomous Technology (Cambridge, MA: The MIT Press, 
1077), p. 284. 

2 Hearsay. 
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as committed to technology as our own: notably, the rapid rate of proliferation of 
new systems, the strong social and economic incentives to use them, and the rising 
level of dependence on computer systems that have vital, nonreversible impacts on 
our lives. If widespread unintended consequences of using computer systems that 
cannot be understood are to be avoided, the problem of Incomprehensibility must be 
analyzed and dealt with now. This thesis Is about the incomprehensibility of many 
computer systems we presently use and on wWch we depend heavily. 

The existing notion of incomprehensibility, as documented in the computer 
science literature as well as In writings from relevant nontechnical fields (such as 
sociology and political science), is highly ambiguous; therefore, my first objective, 
pursued in Chapter 2, is to sharpen this notion and to arrive at a definition from 
which one may more usefully proceed. Some computer scientists have a tendency 
to become so Involved with the details of the newest metiiods of rendering com- 
puter programs understandable (for example, the technique of structured program- 
ming or the avoidance of goto statements) that they seem to forget about more 
general, higher level issues. Conversely, critics of technology's social role, 
although they may appreciate the widespread effects of technological innovation, 
are often hampered by the inability to understand specific technical applications. 
The study of computer system incomprehensibility thus demands an interdisciplinary 
approach, based on an understanding of both computers themselves and the tech- 
nological system of which they are among the foremost representatives. 
Throughout this thesis, I refer to the observations of diverse groups of 
people - computer scientists, philosophers, psychologists, sociologists. It is not 
expected that the reader will be familiar with all the relevant disciplines; therefore, 
I have provided a "list of characters" - a collection of short biographies of most of 
the people on whose ideas I have drawn - following the body of the thesis. 
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In discussing computer comprehenslblllty. It is useful to first consider 
comprehension In a broad context as, for example, in communication between peo- 
ple. This need not be a specialized endeavor; most people have, a strong intuitive 
sense of what it means to understand someone else, and it is this Informally 
learned knowledge that one projects onto one's interactions with computers. Shar- 
pening the general concept of comprehension, as It Is applicable to human communK 
cation, in order to arrive at Its much more specific form, namely as it appears in the 
context of computers, reguires competence In computer science. Insights about 
comprehension, when applied to problem evaluation, system design, programming, 
etc. (all fundamentally human activities), can help establish what it means to 
comprehend a computer system and the behavior of computer systems. 

IncomprehensibiUty Is not a property of a computer system (In the sense 
that color is a property of an orange); rather, It is a derived attribute which is 
dependant upon the conlext In which a system Is used and the criteria according 
to which it is Judged. An aJrHne reservation system might be crystal clear to the 
reservation clerks who use It every day, but largely mysterious to the systems 
analysts who attempt to modify it. In addjaon, incomprehensibility takes on a 
different meaning In relation to the "front end" of a system (analysis of a problem 
and design of a system) than It does to the "beckead" (utilization and mainte- 
nance of a system once It has been Implemented), Difficulties are bound to arise 
either when a system is designed in a haphazard, ad />pc fashion (surprisingly com- 
mon outside the restricted domain of research systems) or when users are forced 
to communicate with a system without some Icnowjedse pf the theory on which Its 
design was based. Indeed* incomprehensible systems, often turn out to be those 
which are based on no well formed theory at all. 

It is important at this point to stress the fact that I am concerned with the 
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incomprehensibility that arises from systems; that is, collections of Interrelated and 
intercommunicating activities, of which computers and computer programs are Impor- 
tant, but not exclusive, components. By the term "computer system" I wish to 
refer to not just computers themselves, but also the people who choose to design, 
maintain, and use them. Computer systems bring Into question much more than Just 
computers and the programs that ran on them; for instance, the nature of the prob- 
lems that we deem suitable for computerized solutions, and the poorly understood 
processes of problem analysis, system specification and design, and programming. 

The kind of incomprehensibility I am interested in does not derive solely, or 
even mainly, from any easily identifiable errors (such as coding errors or obscure 
programming), but from more elusive problems with the way we think about and deal 
with technology in general, and computers In particular. If we apply computer tech- 
nology Inappropriately or Indiscriminately (for instance, If we are more motivated by 
an eagerness to make use of computers than by the actual effectiveness of apply- 
ing computers In a given application), we may end up having difficulty understanding 
the relationship between the original problem and the computer system constructed 
in response to it. In some cases, "problems" are artificially created or tailored to 
make them better suited for applications of current technology. Computer systems 
that arise from such situations can be functionally incomprehensible - incomprehen- 
sible in relation to the problem that a system's users feefteve it is "solving." 

Already in the present discussion, t have turned to the question of how does 
incomprehensibility arise in a computer system. In Chapter 3, I examine factors 
which can lead to the generation of Incomprehensible computer systems. Concerns 
about the process of programming are relevant here: Qerald Weinberg has re- 
minded us that programming is a human activity wfth a psychological component 
which is often ignored, but which significantly affects the quality of programs which 
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are written. The activity of programming a large computer system Is plagued by 
what some people have described as problems of communication. Interactions 
between the diverse groups of people who are touched by computers can be 
stifled by the elitist belief, held by some specialists, that non-technical knowledge 
Is not very relevant in the design of a computer system* st that outsiders should 
not question the appropriateness of applications of technology. Professional Isola- 
tionism helps distort the knowledge that coders, system designers* users* and 
everyone In-between can have of a system (by coders, I refer to people who carry 
out fairly routine programming Jobs which are handed over te t(M*« by pother people 
who are mote Involved with higher level Jobs like the design of a «yste»>. Bfcajrre 
stereotypes of programmers as solitary IndhddueJe who are detached from other 
people are reinforced by the informal «ttitudinat training of software workejrs, which 
tends to discourage curiosity beyond the level of specific, uneonrtectedyprogrein- 

ming tasks. ' 

In addition to relatively tow level problems which may be Inherent In the pro- 
gramming process, there are broader, societal Issue* which have important Impacts 
on our relationship to computation; these issue* are discussed In Chapter 4. As I 
mentioned before, a thorough consideration of the> problems entrained by 
incomprehensible computer systems requires both an understanding of computers 
and a high degree of sensitivity to the social contexts in which computers play an 
Important role. In the modem world, one ought tiot talk about social "problems" 
without tslklng about technology, nor discuss technology without taking Into 
account its social context. 

The present organization of society Is suefc that thete is an air of Inevitabil- 
ity about the role of technology. Our ability to critically evaluate social problems 
and proposed computer solutions to these problems is strongly influenced by what 
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appears to some as our acceptance of the autonomy of technology at the expense 
of our own human autonomy. The perception of technology as an irresistible force 
leads to a situation where the usefulness of the computer is often assumed, even 
when a given application of computer technology Is by many criteria inappropriate} 
where the widespread use of computers is accepted in spits of the fears and mis* 
gtvlngs of many people; where the surface appeal of quick technological fixes for 
pressing problems often causes a redefinition of our problems to make them mors 
amenable to computerized solutions. Present day society uncritically accepts a 
way of life foundetf on technical necessity and shaped primarily by what Langdon 
Winner cams the technical (rational, artificial, productive) mode of activity and 
thought^ Our love affair with technology has always been characterized by blind- 
ness, and the so-called computer revolution Is only the most recent example of this 
technological preoccupation. 

Chapter 5 of this thesis is a discussion of the results of the use of 
incomprehensible computer systems end of the social system In which titey are 
embedded. Some technical results are the inabJHty to ensure adequate reliability 
for many large, complex computer systems, and the resistance of of these systems 
to even minor modifications. Other consequences are extensions of points 1 raise in 
my discussion of potential sources of incomprehenstbiHty; for instance, the obscura- 
tion of the root of a problem as a result of over-rationalizmg the processes of 
evaluating problems and of planning solutions to them. If a computer application is 
viewed solely in information processing terms, there stay be a gap sensed between 
the problem and the computer system which was designed to solve it, but Which In 
actuality attacks onty those symptoms of the problem which were easy to translate 



3 Winner, p. 127. 
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into the form of a computer program. The size of this gap reflects the importance 
of those aspects of the problem which could not be expressed according to reduc- 
tionist criteria, and hence were never reckoned into the design of the computer 
system. Technological elitism, which I cited earlier as a source of McomprehensibH- 
Ity, is strengthened by the existence of computer systems which only a few tech- 
nically trained people can plausibly claim to understand or ba able to maintain. In 

Kenneth Laudon's words, we are experiencing " a legittaiteatlpn of technological 

'experts' at the expense of poets. H 

On a still broader level, an attempt to calculate the costs of living In a tech- 
nologically based society must include some estimate of the personal price that is 
paid - the human suffering. There has been a gradual adaptation of human needs, 
desires, and thought processes to tha% which contemporary technology can explain 
and satisfy. What happens to the sell. Image of people when "all the business of 
life, from work and amusement to love end death, Is seen from the technical point 
of view"? 6 As a result of society's continuing quest for the perfection of the 
machine, some people now seriously question whether we wUJ be able to keep pace 
with our computers. I will examine these and other unintended, but perhaps un- 
avoidable byproducts of the computer "revolution." 

Perhaps the most serious effect of the widespread use of computer systems 
Is our growing dependence on computer systems that we do not understand. The 
use of computer systems that are^ot comprehensible can result in the loss of our 
control over the processes that computers monitor, as we become Increasingly 



4 Kenneth Laudon, review of The Conquest of With Information Processing in 
Human Affairs, by Abbe Mowshowltz, Sc/ertce, ffc3 (September, TOfe), 1111. 

6 Jacques Ellul, The Technological Society (New York: Vintage Books, 1964), 
p. 117. 
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dependant on the Information, and even the decisions, which are the output of the 
computers. In actual approbations Involving large, complex computer systems, peo- 
ple using such systems' output often come to rely on "what the machine says.* 1 
They are effectively In a position of either having to do the work the computer Is 
supposed to be doing, or of having to uncritically accept the output of the com- 
puter. The Inherent complexity of many systems makes it almost Impossible to ade- 
quately explain, predict, or trace their operations. Nevertheless, m using these 
computer systems, one must cross a threshold from reflance to unjustified trust. 
Trust Is deeply embedded with human values, and it Is not at all dear that the pro- 
jection of these values onto man-machine Interactions is appropriate or desirable. 

An unwillingness to depend on computers to aid in Important decision making 
may be deemed evidence of an *antl t e c h no lbgfcaf attitude. Nowadays, an attitude 
of technological optimism and trust appears not ^ need Justifteatlon, whRe techno- 
logical distrust is often greeted \^h hostility. This soelsf pressure against criticism 
of technology Is not Irrelevant to my discussion of inc ompr ehen s ibility. 

I beHeve that there Is a corres pond o n se between our trust in computer sys- 
tems end the purposes of the systems (both our understanding of these purposes 
snd how wen ttie systems actually fttWi them>, that' there Is a discrepancy 
between the real and the perceived purposes ^mehy computer systems, and that 
there Is a Strang relationship between our comprehension of these purposes and 
the appropriateness of trusting the systems. It Is our trust In computer systems 
which makes us so vulnerable to their effects. 

One direct and dangerous effect of trusting Incomprehensible systems Is the 
erosion of the meaning of responsibility for oi^^V^B^tip|is^'.V;rW*» W" i»'. bo r . hadd 
responsible for modern computer systems tiiat have ltter«uV evolved into their 
present forms and that simply cannot be said to have authors? Already, some 

14. Chapter ts Introduction 



people are suggesting that the computer be responsible for its share of the deci- 
sion making load, but It Is far from clear what possible meaning computer responsi- 
bility could have for the people affected by machine made' decisions. Some advo- 
cate the idea that systems retain control over ongoing processes. In practice, 
Incomprehensible systems become autonomous, unchallengeable authorities to whom 
a society of users abdicate responsibility. According to Joseph Welzenbaum, 
"... responsibility has altogether evaporated. Mo human far any longer responsible 
for 'what the machine says.' Thus there can be neither rigrrt £ ftbf wrong, no ques- 
tion of Justice, no theory with which one can agree br disagree, and finally no basis 
on which one can challenge 'what the machine says.'" 

Certainly I do not wish to leave the reader with nothing but dismal pro- 
nouncements about the impossibility of individual action against an autonomous 
technology. In Chapter 6, I discuss alternate courses of action In Hie face of 
Incomprehensible computer systems. First, I consider various technical "solutions." 
These Include verification proofs, reliability studies, and modern programming prac- 
tices like structured programming. In the course of analyzing these and other 
means of injecting understandability into computer systems, it becomes clear that, 
for the most part, they attack only the incomprehensibility of computer programs. 
However, the enforcement of structure on the product (the program) does not 
necessarily enforce structure or comprehensibility on the process which created it 
(the design of a system), and it is the larger computer system that is the concern 
of this thesis. 

I believe that system Incomprehensibility is fundamentally not a problem in 
the engineering sense of that word, and that the most Interesting kinds of 



6 Joseph Welzenbaum, "On the Impact of the Computer on Society*" Science, 
176 (May 12, 1972), 613. 
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Incomprehensibility are not to be dealt with by technical mean*. Rather, we must 
make a conscious effort to widen our perspective beyond a narrow professional 
one. The social consequences of using irw:o mpfe i h enslbie computer systems have 
direct impacts on the meaning of responsibility, the ethical and moral burdens of 
computer scientists, and the self Image of all peopja. It Is necessary to reckon not 
only the technical, but also the human costs of the tachnpfoglGal system; In Lewis 
Mumforcfs words, to ask "not what is good for science or technology, . . . , but 
what is good for man ... ." 

There are positive steps which can betaken .to, avoid the potentially 
dangerous effects of the existence of vital computer systems that are functionally 
incomprehensible. As Vice-Admiral H. Q, Rlckpver has pointed put, a good beginning 
may be made by reflecting on whether or not everything hailed as progress actually 
contributes to happiness (or whatever else the reader believes sustains human cut- 
ture), and remembering that we alone are responsible for our technology. 



7 Lewts Mumford, "Authoritarian and Democratic Technics," in Technology and 
Culture, ed. by Meivbi Kranzberg and William H. Davenport (New York: Schocken 
Books, 1972), p. 58. 

8 H. 6. Rickover, "A Humanistic Technology," In Technology and Society, ed.^ 
by Noel deNevers <USAt Addteon-Wesley Publishing Company, 1 072), pp. 22-23. 
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Chapter 2: Understanding Computer Systems 

There can be no total understanding end no absolutely reliable 

test of understanding. Joseph Weizenbaum 

I understand*" ELIZA 

It Is acknowledged within the computer science community that the 
incomprehensibility of computer systems is a major problem associated with the 
rapid proliferation of large scale computer applications. Nevertheless, incomprehen- 
sibility has proved to be a most difficult concept for both computer scientists and 
social scientists (working from technical and non-technical perspectives) to define. 
Computer specialists hint at the issue of incomprehensibllltywhen they discuss the 
"software problem," which Is really a whole collection of problems that make com- 
puter programs, particularly large ones, intractable. There seems to be something 
inherent in large computer systems, perhaps it is the complexity of such systems, 
which fosters incomprehensibility. A large computed system Is worked on by so 
many people over such a long period of time that there is finally no group of people 
who can be said to be Its authors or who understand it in any usefuf sense; for 
example, well enough to guarantee its reliability. 

It is probably premature to Jump right Into a discussion of computer 
Incomprehensibility before saying something about comprehensfon In the context in 
which we are most familiar with it - that of interpersonal communication. Our feel- 
ings of understanding another person and of being understood by others are not 



1 Joseph Weizenbaum, Computer Power and Human Reason: From Judgmanljo 
Calculation (New York: W. H. freeman, WCb p.J03. = , _ v , , S( ^ 

2 A computer program, published in 1968, which makes poss|>ie<oertaln k|nds 
of natural language conversation between man and computer. 
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formally learned, but are instead the results of Intuition. Some aspects of this 
intuitive knowledge seem to be shared among all people by means of unspoken 
channels; communication ts by definition an act of sharing. The assumptions which 
make human communication work (to the extent that it does work) most often 
remain unstated. The incomprehensibility of computer systems is such an elusive 
notion because our sense of what it means to understand is not externalized when 
we change our frame of reference from that of people to that of machines, and 
begin to talk about understanding computers. 

2.1: interpersonal Understanding 

As a result of our Interactions with other people, we arrive at an informal 
definition of understanding, according to which understanding Is a function not only 
of words that are spoken (or thoughts that are communicated In other ways), but 
most importantly of the speaker and the listener, who bring something of them- 
selves to any exchange with another person. Comprehension, as impHcltr/ defined 
in human relationships, is to a large extent founded on shared experiences and 
values. Interactions between people nearly always require some element of faith, 
based on our trust that the other person will "know what I mean." This trust is 
justified only because "all people have some common formative experiences .... 
There Is consequently some basis of understanding between any two humans simply 
because they are human." 4 We find that our trust Is best rewarded by people who 
see the world as we do, and in particular by those people who have had expert- 



^he Oxford English Dictionary (Oxford: Oxford University Press, 1033), 
p. 099. 

*Welzenbaum, Computer Power and Human Reason: From Judgment to Cafct/- 
tatlon, pp. 102-193. 
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ences similar to ones that we have had and that were important to us. An under- 
standing between people Is not something carelessly acknowledged, it is only 
when we feel that someone else's Interpretation of the world 4s sympathetic to our 
own that we are ready to believe that that person can understand us. 

One of the most Important observations that can be made about human 
understanding is that there are Insurmountable Mmtts to the level of communication 
that can be achieved. 

Since, in the last analysis, each of our lives Is unique, there Is 
a limit to what we can bring another person to understand. 
There is an ultimate privacy about each of us that absolutely 
precludes full communication of any of our ideas to the universe 
outside ourselves and which thus lsdtetees?ee*h!«n« of urn from 
every other noetic object in the world .... 

TO knew with certatoty 4hat&#<i>e»een iwderstood what 
has been said to him Is to perceive his entire belief structure 
and triat is equivalent to sharing hts enlimWegeixpejtencov 

Attempts to communicate with other people are largely sets of faith, substantiated 
in part, though never entirely, by a common language, social background, environ- 
ment, experiences, etc. I wish to stress this point - tte* signlftcance of a common 
humanity m human understanding - because it is precisely tills common humanity 
that Is not part of our "relationships" wltii computers* an* thus ceimot play any role 
in an understanding between man and machine. 

2.2: Comprehending Computers 

An emphasis on the human element of Interpersonal communication leads 
directly to the belief that understanding must take on a very different meaning in 
relation to Interactions between men and machines than In relation to Interactions 
between people. Just what It is that characterizes this difference is not immedi- 



6 lbid., p. 193. 
Section 2.2: Comprehending Computers 1 ^- 



ateiy apparent, since there are a number of things that we can mean when we say 
we understand a computer system. The nature of our understanding of computers 
depends on the context in which we make use of them (which may be as a neutral 
storehouse of information or as a decision-maker on which we depend heavily) and 
varies hi relation to the many different elements that combine to form a computer 
system (from computers themselves and computer programs to the .human beings 
who Interact with them, and even the stated and the perceived goals of a system). 

2.2.1: Program Output 

The simplest interpretation of understanding in the context of computers is 
that of understanding the output of a compute/, One means of achieving this level 
of comprehension would be tracing the operations of a computer progranKs) that 
generates particular output that we are Interested in and wish to understand- 
Unfortunately, there are strong pressures that can effectively preclude such direct 
Involvement in computer applications. 

There are numerous social factors which subtly encourage the use of com- 
puters and discourage any serious criticism of them. One aspect of the "American 
way" of life is the use of the latest taohnotaatoal "breakthroughs"; Norbert Wiener 
gives a compelling discussion of what he terms "gadget worshiping": 

Of the devoted priests of power, there are many who regard 
with impatience the limitations of mankind, and In particular the 
limitation consisting in man's undependebfflty and unpredicta- 
bility 

In addition to the motive which the gadget worshiper finds 

for his admiration of the machine in its freedom from the human 
limitations of speed and accuracy, titer e M* en«dmotive which It 
is harder to establish in any concrete case, but which must play 
a very considerable rote nevertheless, ft m the desire to avoid 
the personal responsibility for a dangerous or disastrous deci- 
sion by placing the responsibility elsewhere: on chance, on 
human superiors and their policies which one cannot question, or 
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on a mechanical device which one cannot fuHy understand but 
which has a presumed objectivity .... 

Once such a master becomes aware that some of the sup- 
posedly human functions of his slaves may be transferred to 
machines, he is delighted. At last he has found the new 
subordinate - efficient, subservient, dependable In his action, 
never talking back, swtft; and not demanding a single thought of 
personal consideration. 

Today, gadget worshiping is often Implicit in the use of large computerized informa- 
tion systems. In the minds of many people and perhaps of society as a whole, an 
unwillingness to make use of current technology bespeaks a backwards attitude 
and even a lack of support for good old American Ingenuity. People who suggest 
checking up on computers (which surprisingly many people still think of as the 
machines that never make mistakes) are thought of as simply not keeping up with 
the times, and because of this they may not be deemed suitable for professional 
advancement. Few people can be expected to stand up to the social pressures 
exerted by the workplace (In the form of professional recognition) and by a society 
in which the omnipresence of technology is almost completely accepted. 

Wholly apart from the ways in which a computer system may directly and 
tangibly benefit a company, computerizing one's business operation is a status 
enhancing act; computers are image builders. It is not difficult to find serious busi- 
nessmen (and even more frlghteningly, serious computer science researchers) who 
use computers at least to some extent simply because everyone else uses them, 
even though this justification for employing computers reduces to nothing more than 
the age old syndrome of "keeping up with the Joneses." In a recent public talk, a 
bank executive admitted that some of the portfolio counselors In his department 
display computer output primarily to Impress clients fcy enhancing the Image of the 



6 Norbert Wiener, Qod and Go/em, Inc. (Cambridge, MA: The MIT Press, 1064), 
pp. 63-56. 
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bank. In some cases, It Is not much more than such Insubstantial reasons that 
underlies the large investments of time and money needed to build a computerized 
business system. Given the predominantly social role that some systems fulfil, it is 
not surprising that understandabHity merits a tow priority in these situations. 

In addition to social considerations, there are strong economic factors that 
have a major influence on a buslness's use of computers. The pressures of 
economic competition can create a situation thst Is far more dangerous than mere 
wastefulness suggests. The real danger lies in the fact that constraints of time 
and money prevent most users of computer generated output from verifying the 
correctness of the output that they must often depend upon in making decisions. 
John Kemeny, president of Dartmouth College, remarks that "It is a simple economic 
calculation that a man who earns $26,000 a year cannot afford to spend a week 
doing by hand something that a computer can do in five minutes." Unfortunately, 
the opposite may be true. Particularly in circumstances where vltsl decisions are 
being made (e.g., military command and control applications), what we cannot afford 
is an unjustified dependence on Information and decisions output by a computer. 
Nevertheless, given the existing emphasis on cost-effectiveness In the business 
world, understanding computer systems by directly monitoring their output is fre- 
quently not feasible. 



Laurence Rebieman (Vice-President, First National Bank, Boston, Mas- 
sachusetts), "Computers and the Workplace," lecture sponsored by; Mtf Program for 
Science, Technology, and Society, March 7, 1978. 

8 John Kemeny, Man and the Computer (New York: Charles Scrlbner's Sons, 
1972), p. 107. 
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2.2.2: Theory of Behavior 

A more realistic way of satisfying ourselves that we understand a computer 
system (and the same way in which psychologists often satisfy themselves that 
they understand some aspect of human beings) is to have a theory of the behavior 
of the system, to which we can turn in order to explain and to verify the output of 
the system. Such a theory can prove extremely useful la understanding the opera- 
tion of a computer system and not just the Individual programs that comprise it. 
This higher level of explanation is more economical than a detailed account of com- 
puter programs (perhaps in the form of commented program code or lengthy docu- 
mentation), and It aWows us to predict the output of a program tijat ,s tr »e Implemen- 
tation of the theory and to establish when that Rrogra* malfunctions. It will still be 
possible for a sufficiently complex system to^surprjse uj,.but this need not invali- 
date our theory; rather, non-standard system behavior may act as. a test of the 
validity of a theory, and the theory itself can serve as a check of the correctness 
of the system. 

There are certain trade-offs exacted In return for the convenience and the 
security of having a well founded theoretical explanation of a computer system. In 
defining our understanding of a computer system by our understanding of the theory 
behind the system, we are enforcing soma ^^oTftfr«?# baleen the system and its 
users. People who make use of a computer system may come to think of It more In 
terms of the theory that constitutes a behavioral abstraction of the system than of 
the underlying hardware and software. Thus, although the existence of a theoreti- 
cal foundation may make clear when a computer system ifcjes sometiilng wrong, ft 
will require a shift in perspective to knotf how to modify the underlying «Ode to 
prevent future occurrences of the aberrant bshavior of tile system. The code 
Itself may be largely incomprehensible, so that modifications may very well damage 
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system reliability. In any case where a system "bug" 8 is discovered, but either 'its 
source persists in being elusive or the costs of modifying 1he system are too great, 
users of the system are left with a choice of abandoning the system or adapting 
their behavior to the system's quirks. Alt too often, people come to depend on a 
computer system too much to consider working without It and reverting to older, 
pre-computer ways Of doing things, Thus, It is not unusual to see ah adjustment of 
human needs so that they may better correspond to that with whTch existing com- 
puter systems can reliably (which is to say, without much Intervention on our part) 
provide us. 

When a computer system is described in terms of a theory of its use, the 
problem of comprehending the system can be viewed as a collection of problems 
that involves understanding first the theory Itself arid thin the relationship 
between the theory and the system.* The most easily comprehended systems are 
those that deal with some widely understood, unambiguous subject matter. An 
example of a system that is extremely large and complex, but nevertheless under- 
standable in these terms, is the MACSYMA computer System for solving problems in 
the integral calculus. The mathematical theory on which this system is based Is 
sufficiently well defined that it is not open to much dispute nor subject to major 
modifications. If MACSYMA makes a "mistake,* the User should be able to detect 



An unwanted and unintended property of a program. 

'°ln other words, the system is a model of the theory: "... if we view the 
theory we incorporate Into a program as an. un/nter/w^ted theory, we are free to 
view the computer's behavior as satisfying one interpretation of trie theory, so that 
any pr ogr am me d computer can be i viewe< as^bisjtar^latln^ (on one interpretation) 
the theory incorporated In its program, the tokens of computer behavior tiiat on 
one interpretation are uttered descriptions of the beha^viof of some other entity In- 
stantiating the theory, can on another Interpretation be viewed as themselves fn- 
stances of behavior predicted i,W ■.<■ >4l*fc v MM^pry^ ^ X3m^ C. Dennett, 
Brainstorms - Philosophical Essays on Mind and Psychology, (USA: Bradford Books, 
1078), p. 194n. 
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the error by referring to this underlying theory. Computer system users need some 
knowledge of the theory according to which a system has been constructed in 
order to avoid a situation where a system is set up as an unquestionable authority 
and to justify the trust that is implicit in many interactions between computers and 

people. 

Once we have accepted the validity of a theory, there Is another aspect of 
understanding a computer system: we must convince ourselves that the system is 
a valid implementation of the theory. Idea4ly (if our understanding of the theory is 
nearly complete), we can subject a system to a number of sample cases designed 
to exhaustively test it. If a system fails to crack under the stress of extensive, 
worst-case testing (assuming we can always implement such testing), our 
confidence in it will be at least partially established. None of this, however, 
touches the question of when is it appropriate for us to rely on a computer system. 
The appropriateness of trusting a system in a given application depends on much 
more than even the most thorough testing; I will have more to say about this issue 
later in this paper. 

The reason most often given to justify the need for computers is the com- 
plexity of present day society - the immense quantity of information that must be 
processed and the variety of connections between data that must be stored and 
analyzed. If a theory captures the essential means and ends of a problem we set 
out to solve with the aid of computers, then expanding our view of a computer sys- 
tem from the level of its actual components (such as physical devices and 
software) to that of the theory on which It is based can be a great aid in coping 
with the complexity of many modern systems. Enormously large, complicated sys- 
tems like MACSYMA deal with complexity by giving up local understanding of the 
details of the system, while maintaining a more global understanding of the goals of 
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the system and how it goes about achieving them. 

We may not be able to guarantee high level understanding of a system if it 
is not founded on robust theories; however, the existence of a well founded 
"theory of use" can enable us to comprehend a large computer system without hav- 
ing to keep track of the numerous details of its operation. Most of MACSYMA's 
users can understand the system well enough for their purposes without being 
aware of the software that interprets and carries out their requests. We may 
relinquish direct understanding of many details of a computer system to accomplish 
very large or complex tasks that are better understood In terms of a concise 
theoretical explanation than in terms of a series of computer programs. Some com- 
puter applications are simply too large in scope to be usefully understood in detail 
by anyone. It is important to note that the difficulties raised by the complexity of 
such applications are surmounted more by our own understanding of the problem 
domain (which may be in the form of a well constructed theory) than by the partic- 
ular computer system that may implement a solution to the problem. 

2.2.3: Criteria and Context of Use 

Our understanding of any computer system depends on the context in which 
we make use of ft and the criteria according to which we judge it. We may 
comprehend a particular system well enough to feel confident of its reliability in 
normal usage, but not wen enough to risk making changes to it. It seems reason- 
able to expect the computer systems we use to demonstrate high standards of 
reliability, maintainability, flexibility, modifiabHity, etc. Many systems satisfy some 
of these criteria, but ft is questionable how many systems achieve enough of them 
to really satisfy their users or to justify their continued use. 

It is the job of system designers to decide which characteristics will be 
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most vital to the operation of a given system, and to make clear the extent to 
which certain criteria will be satisfied by the system. It Is then up to a System's 
users to be aware of the strengths and weakness of the system (just as we 
accept limitations in people), and not to attempt to Interact with the System in 
ways that are contrary to Its "character." In other words, people who deal in any 
context with computer systems should consciously decide In what ways it is 
appropriate for them to Interact with a computer. For instance, It seems obvious 
that a system that is not designed to reliably support modifications (such as the 
federal social security computer system) should probably not be modified once it is 

in operation. 

Unfortunately, simplistic guidelines like the one mentioned above are not 
likely to be of much use in the complicated environment that surrounds a large com- 
puter system. To begin with, modiflabtlity and other important properties of any 
computer system are In general not "built Into" a system in the process of system 
design or Implementation. 11 In practice, the major influences on the design of a 
computer system often turn out to be the cost of the system and externally 
Imposed deadlines for its completion. Computer professionals must at times settle 
for getting a system up and running with the smallest respectable amount of test- 
ing and documentation. Attributes like reliability may not be investigated until a sys- 
tem is in ust and/evelf^wo^, iintil people i Save betfun to : depiha on It; at-this 
point, it may prove necessary to make changes to a system whose reliability in the 
face of modifications is highly questionable. 

It is often the case that the extent to which a computer system exhibits 

11 An important exception to this statement is the following: programmers 
and system designers often leave "hooks" Irt the* programs, so that foreseen ex- 
tensions can be made more easily and more reliably hi the future. However, one's 
vision of what hooks should be left Is often severely limited. 
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such features as modifiabillty and reliability is revealed while the system Is in use, 
and that our understanding of a system Is formed by means of "experiments" over 
the course of time. Since tiie consequences of computer systems can be both non- 
trivia! and irreversible, it is of the utmost importance for us to consider who are the 
real experimental animals. Clearly, It is not always possible or advisable to deter- 
mine a system's correctness or resistance to change by, for Instance, Implementing 
some crucial modifications and watching the results of the changes on the actual 
operation of the system (as opposed to a test run). 

The program code of commercial systems can easily develop into a patch- 
work of quick fixes for unforeseen difficulties. Clever but obscure technical 
patches may be hastily applied to the original programs In response to pressures 
from corporate management for minimal production time and system cost, and from 
users for minimal system down time. Tracking down a difficult "bug" can lead to 
insights which, if perceived before the system was implemented, would have 
resulted in a different system organization than the existing one. However, once a 
system is in operation (in fact, once its Implementation is under way), restructuring 
it according to new knowledge is usually not feasible. Besides the obvious 
economic factors which effectively preclude serious reorganization, there are 
psychological influences: after the interesting, challenging work of locating a bug is 
finished, the Job of restructuring a system may appear comparatively tedious. 
Thus, even if there is no pressure to work quickly, the temptation to patch up a 
system, rather than to attempt to restructure it, is strong. Unfortunately, It is fre- 
quently impossible to predict the behavior of a computer system that consists of a 
collection of patched-together programs. 

Particularly in the case of systems with unclear or potentially dangerous, 
nonreversible consequences, issues like reliability should be analyzed weH before 
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the system Is put together. Such an analysis should be made from a non-technical 
viewpoint as well as from the standard technical viewpoint according to which some 
computer systems are already Judged. A given reliability factor may warrant the 
use of one system In an academic application, but not the use of another system 
as an air traffic controller, and the methods of making these Judgments are not 
exclusively those of a formal technical study. 

The most thorough analysis of a computer system would have to take into 
account subtle issues, such as the degree to which users of a system are likely to 
depend on it and the possible consequences of this dependence. Issues like these, 
Which bring into play the people who use, depend on, and are otherwise affected 
by computers, are likely to be more inscrutable than computer systems themselves. 
Although there has been recent evidence within academic circles of sensitivity to 
the "human factors" of computer systems, few critical, non-technical sentiments 
have filtered through enough economic and social channels to be evident In the 
design and use of existing computer systems. 

Our knowledge of the extent to which a computer system measures up to 
certain important standards (which will in all probability be tested at some point In 
the course of its lifetime) is a major factor in our understanding of that system. A 
lack of attention to criteria such as those I have talked about here will severely 
limit the ways in which we can reliably, comprehensibly interact with computer sys- 
tems. Shallow levels of understanding are hardly enough to warrant the high 
degree of trust that many people place in computers. The advisability of exercis- 
ing great caution in constructing and using bigger and better computer systems 
seems obvious (particularly in the unexampled, poorly understood areas of some 
current computer science research), but some of the most WgWy respected com- 
puter scientists proceed in their work largely by employment of the method of trial 
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and error, without much consideration of the ramifications of their work. 

As a result of the previous discussion, we can see that the criteria by which 
we judge a computer system are closely related to the context In which we use it. 
Different, and probably loss stringent crrteria would be appropriate in the considera- 
tion of an airline reservation system than of a military command and control system. 
Context and criteria of use help determine the risks and benefits of using a com- 
puter system - what might be called the system*s "degree of vitality" - on which 
any discussion of the appropriateness of utilizing the system should be based. Ulti- 
mately, a computer system's degree of vitality depends on the' ways in which the 
system may affect people. 

Finally, the context of our use of a computer system has something to do 
with our perception of the relationship between the i system and the problem It pur- 
ports to solve - the extent to which a computer system attacks the problem we 
actually want it to solve. As I wlli discuss later in this thesis, computer systems 
designed to solve social "problems" (for example, the faBure of most modem school 
systems to educate or to motivate their students) often miss their mark. Many 
times, this is so not because of poor programming, but because we do not under- 
stand the nature of the problem. In applying computers to a relatively superficial 
symptom (such as the current shortage of teachers) of a persistent social problem, 
rather than coming to grips with the real source of flie difficulty (which may or may 
not be "computable" 12 ), we are taking the easy way but of the problem. We may 
understand how our system deals with the problem symptoms on which we have 
concentrated our attention, but we will probably not understand why the system 



12 By "computable," I do not mean to refer to any technical definition of 
what can and what cannot be repr ese n ted by a computer algorithm, but rather to a 
more Intuitive sense of the kinds of problems to which a computer system may ao- 
propr/ate/y be applied. i- 
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does not solve the original problem (and It la likely that It wtH not solve It), since 
we never acknowledged the root of the problem - the human sources of the 
difficulty - in the first place. The utilization of computers as -patch" solutions to 
poorly understood problems can be the source of a subtle kind of Incomprehensibil- 
ity that is extremely difficult to detect. The incomprehensibility of such systems 
derives from the absence of a well thought out explanation of the difficulty at 
hand, preceding the design of a computer system to resolve that difficulty. When a 
system Is constructed to model or otherwise refer to a human activity, Its design 
should reflect first and foremost an understanding of that activity and not Just an 
attempt to operationally replicate It. 

2.2.4: Front End Versus Back End 

In discussing computer systems, we must make dear whether our viewpoint 
is that of the designers of a system or of Its users. Understanding takes on a 
different meaning with respect to the design and generation of a computer system 
(what I will refer to as the "front end" of the system) than It does with respect to 
the use of a system (the "back end"). 

A system's front end comes Into being before the system Is even built, with 
the basic decision to use computers to deal with a particular problem. Before the 
technical design of the system can begin, an understanding of the problem must be 
arrived at. Often, the people who have the most complete knowledge of the prob- 
lem domain are not the same people who set up a computer system in that domain. 
Thus, In addition to difficulties Inherent in the problem being analyzed, there is likely 
to be another stumbling block - that of communication between the technical spe- 
cialists who set up the computer system and the non-technical workers who provide 
input into the specifications of the system and who will eventually make use of it. 
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An obvious way of dealing with this communication predicament is to include 
both computer engineers and system users in the design of a system. Unfor- 
tunately, there is often considerable difficulty in bridging the gap between different 
perspectives of a computer system (that of the engineers and that of the users, 
for instance). This difficulty cannot necessarily be resolved technically (for exam- 
ple, by developing a sufficiently high level programming language that serves the 
needs of the various groups of people who use a computer system), since it arises, 
at least in part, from discrepancies in the ways in which different people interact 
with and think about computers. 

Besides difficulties with communication, there are other difficulties that can 
preclude a well organized front end of a system. Consider the fact that computers 
are often introduced Into situations which have never been overtly organized along 
rational lines. With the suggestion of computerized operations, an existing, informal 
system must be converted Into a form that is suitable for representation by com- 
puter programs. In applications outside the realm of academic research (/.e., appli- 
cations that are often characterized to a larger extent by "sloppy " human interac- 
tions than by well defined rules), the task of rationalizing the work often proceeds 
in a haphazard, ad hoc fashion. Moreover, it is often the case that the work itself 
is redefined In terms of the tools (in the present discussion, computers) that are 
available. 

One case study which exemplifies the difficulty of converting pre-computer 
activities into computerized systems is the design and implementation of the Bank 
of America computer system in the late 1950's. The designers of this system 
quickly discovered that there was no organized system of rules and regulations 
that completely governed the activity of banking. Despite the large scale and the 
obvious complexity of this system, transactions that were not strictly routine were 
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often handled Informally, many on a case-by-case basis. In this way, bank person- 
nel were able to take Into account unusual aspects of a cllenf s situation and, to 
some extent, personalize their decisions. As the banking system grew to large pro- 
portions, no formal policy was ever necessary because the system worked. 
Because life gives feedback, contradictions can be resolved Informally (sometimes 

we call this leniency). 

It may not always be in the best Interests of a system to formalize Its deci- 
sion making processes. In the case of the Bank of America system, the decision to 
utilize a computer system necessitated the establishment of a system of rules that 
explained banking procedures; as a result, the banking system was changed. Per- 
sonalized decision making was discouraged as a result of the Introduction of compu- 
terized banking. Contrary to Vlce-Admlral H. G. Rlckover-s dalm that "of technology 
it can be truly said that it Is not 'either good or bad, but thinking makes It so,'" 
the computer did not remain neutral with respect to the activity of banking. The 
presence of the tool substantively changed the activity that was computerized, 
even though this was not the original intention. 

Haphazardness in the organization of the front end of a computer system Is 
bound to propagate to the back end, as a result, a system's users will experience 
varying degrees of difficulty and confusion In utilizing the system. When a com- 
puter system is introduced to replace an activity that makes significant use of non- 
standardized channels of communication, important characteristics of that activity 
are liable to be lost in the process of converting it into a computable form. Users 
may find that different decisions are being made than had previously been the 



13 Vice-Admlral H. 6. Rickover, "A Humanistic Technology," In Technology and 
Society, ed. by Noel de Nevers (USA: Addteon-Westey Publishing Company, 1072), 
p. 21. 
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case, that seemingly small mistakes can have nontrlvial, unforeseen consequences, 
and that exceptions are no longer as easy to handle as they once were. More 
significantly, users may be led to depend on a computer system that acts according 
to a patched-together theory of their work that Is entirely unclear to them. If 
there Is no well formed theory of use behind a computer system, then the operation 
of the system is little better than experimental; in such cases, dependence on a 
computer system seems highly irresponsible. 

2.2.5: Systems 

Before going any further in this discussion, I should say something about the 
important distinction between computer programs and computer systems. Much of 
the incomprehensibility on which concerned computer scientists end up focusing is 
that of computer programs and, to a smaRer extent, of the processes of problem 
analysis, design, and programming Itself, which culminate in the generation of pro- 
gram code. We are exercising extremely narrow vision In restricting our view of 
computer system incomprehensibility to the incomprehensibility of one part (albeit 
the most logical and well understood part) of computer systems - computer pro- 
grams. 

Butler Lampson, a senior research fellow of the Xerox Corporation, has 
stated his belief that "it is the source text that completely defines the [computer] 
system." 14 It is not surprising that his ideas about making computer systems more 
comprehensible involve schemes for things like building structure Into program code 
(through greater attention to program hierarchy and interfaces between programs). 
Lampson claims that by making explicit all changes to code (for instance, by 



1 hotter Lampson, "Building Programs," MIT Laboratory for Computer Science 
Distinguished Lecturer Series* May 1, 1078. 
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keeping track of the editor in some well defined way), we wHI be able to explicitly 
capture "the nature of the changes," and thus figure out what Is going on. 
Lampson's concerns about the dynamic nature of program evolution are valid ones. 
However, his persistence in keeping everything but computer programs themselves 
out of his treatment of incomprehensibility Ignores the fact that a complete system 
is more than Just computers and the other machinery associated with their opera- 
tion. 

Even assuming that there existed unambiguous, organized methods of making 
computer programs understandable (and this in Itself presupposes that we know 
and agree upon what it means to understand a computer program), a computer sys- 
tem cannot be rendered comprehensible simply by "building" understandablllty into 
Its software. For instance, I have already mentioned the Importance of taking Into 
account the context of use of a computer system in any discussion of the system. 

. . . most large systems actually In use today and on which 
people depend . . . simply are not logically deterministic sys- 
tems In any useful sense of the term. It no longer makes sense 
to speak of such systems as having a state at a particular 
time, or of their programs as If these were concrete texts hav- 
ing an existence independent of the detailed circumstances In 
which they are used. 

T. D. Sterling, writing in a recent article on the social Impacts of computing, 
expresses my own desire for a wide view of computer systems; he repeats KHng 
and ScacchFs definition of a computerized "package": "not only devices (e.g., 
hardware, software, and systems protocols), but also a diverse set of skills, organi- 
zational units to supply and maintain computer based services and data, and sets 
of belief of what computing is good for and how it may be used efficaciously." 



15 Joseph Weizenbaum, "Human Choice in the Interstices of the 
MegamacWne," pp. 12-t3. Lecture presented at the IF1PS Conference on "Human 
Choice and Computers," Vienna, Austria, in June, 1970. 

16 T. D. Sterling, "Consumer Difficulties with Computerized Transactions: An 
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Like these people, by referring to computer systems I mean to suggest something 
with many dimensions, some of which have found no place in purely technical dis- 
cussions of computer incomprehensibility' The all-encompassing nature of computer 
systems is what makes it so difficult to define and to deal wrth the incomprehensi- 
bility that derives from them, but the frequent tendency to shy away from con- 
sideration of the societal aspects of computers has helped entrench the 
incomprehensibility of some systems. 



Empirical Investigation," Communications of the ACM, 22 (May, 1978), 284. 
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Chapter 3: Sources of Incomprehensibility 

Evidently technological accomplishment has become a tempta- 
tion that no person can reasonably be expected to resist. The 
fact that something is technically swea* 1* en**gb to warrant 

placing the world In jeopardy. 

Langdon Winner 

It is hard to realistically deny the pervasiveness of incomprehensible com- 
puter systems In modern society. Most of the computer setanttots with whom I 
have spoken have ready examples of their pet Incomprehensible systems (large 
operating systems are favorite Choices). In discussing incomprehensibility, many of 
these people smile knowingly at me as they remark that it certainly Isn't difficult to 
find Instances of baffling computer systems. What l fWd partlculerty upsetting 
about these conversations is the fact that the comments are made from a profes- 
sional viewpoint; that is, the systems to which these peepi* refer are felt to be 
incomprehensible to computer specialists. What, then, is the position of all the 
non-technical people who make use of and are otherwise affected by large sys- 
tems; how are these people to understand a computer system that the "experts" 
have called incomprehensible? If the seriousness of the problem of Incomprehensi- 
bility is acknowledged by computer scientists* why is It that incomprehensible sys- 
tems are so widely used today and that many systems currently being constructed 
are likely to emerge Incomprehensible In some sense of the word? 

in this chapter, I consider the question of how does Incomprehensibility arise 
in a computer system. Like most questions that Involve human Interactions and 
needs (and those dealing with computer systems do), this one does not have a sin- 
gle, well defined answer. Rather, Computer system Incomprehensibility is a problem 



1 Langdon Winner, Autonomous Technology (Cambridge, MA: The MIT Press, 
1077), p. 73. 
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that brings together a host of different issues. The "reasons" that we may come 
up with to explain the existence of incomprehensible systems depend on our point 
of view - that of a researcher, pregramater, designer, businessman or other user, 
social critic, etc. For the purposes of this discussion, incomprehensibility Is best 
considered from all these perspectives. 

3.1: The Prograinmina Process 

Computer programs are among the most obvious components of computer 
systems, so we may begin an analysis of the source* of incomprehensibility at the 
level of computer programming, I have previously discussed some problems of com- 
munication that beset many large programming injects. Ijlost software workers 
are aware that there are a variety of difficulties with virtually every stage ,of the 
construction of a computer system, and that these difficulties confound program- 
mers working both individually and as part of : : m larger endeavor. 

On a personal level, programmers aj-et viewed fej£ much of society as rather 

peculiar people. Computer scientists must cope with unproductive stereotypes that 

emphasize what might be catted the "hacking mentality.'? 

. . . the hacker is "without deflnlte purpose": he cannot set 
before himself a clearly defined long term goal and a plan for 
achieving it, for he has only tee^nto^% not k ^^^ e ^ge . He has 
nothing he can analyze or synthesize; in short, he has nothing 
to form theories about Hte^mJkW,,: ^ie4|N«lf>t^a«^^^«MtaNfMBSi>-' .«>V«n 
disembodied. It is simply not connected with anything other 
than the instrument on which it may bSrexercised. 

Programmers are sometimes perceived as having Uttte purpose other than to spend 
time with computers; they may be seen as solitary individuals whose strong attach- 
ment to machines is proportional to their detachment from and tack of experience 



2 Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to 
Calculation (New York: W. H. Freeman, 1076), p. 118. 
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with other people. 

Professional Isolationism is encouraged through the training of software work- 
ers, which single-mlndedly emphasizes technical skills. Langdon Winner notes that 
engineers typically regard themselves as "merely problem solvers": 

"Tell us the problem," they demand. "We wlH And a solution. 
Thafs our job. But you may not presume to question the natwr* 
of our solution. You are not a member of a technical profession 
and, therefore, know nothing of relevance. r ?l# you Insist on rais- 
ing questions about the appropriateness of the means ^re dev- 
ise, we can only Conclude that you are antttechnQtagy-" 

The personal identification of programmers with their programs, which occurs in 
many existing set-ups, does not support prodticthieL communication between work- 
ers. In addition, workers in teehrtotogieal fleMs ate subtly taught to accept seg- 
mentation in the work structure that surrouadpr them. Rfogfammers are discouraged 
from asking questions or exploring issues beyond their am l gn ed teaks; they are 
"deprived of all but the most narrow skills and of an -uederstendlng of how their 
work ffts Into the work process as a whole." 4 Coders may fee forbidden to use any 
programming techniques but a few rigidly defined ones, so that their program code 
emerges as little more than "a standardized product made In In a standardized way 
by people who do the same limited tasks over and ever without knowing how they 
fit into a larger undertaking." 6 

Even on a more organized level (e*., that of the management of a software 
project), the activity of programming is often viewed in an impersonal way. In his 
book, The Psychology of Computer Programming (which ie r incidentally, one of the 
most widely read books about computer programmers), fiereid Weinberg comments 

3 Winner, p. 1 1 . 

4 Phillp Kraft, Programmers and Managers: Tbe R+utl nidation of Computer 
Programming In the United States {Htm York: Sprlnger-VerJag, 1977), p. 83. 

6 /b/c\, p. 59. 
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on the widespread lack of attention devoted to the informal, interpersonal (as 
opposed to formal, organizational) structure of software projects and to the social 
(as opposed to physical) environment that supports the work of programming. The 
fact that Weinberg finds it necessary to stress the human element of programming 
is Indicative of a general lack of understanding of and concern for programmers and 
the Interactions between them, that is not uncommon in managerial circles. Philip 
Kraft, too, tias much to say about the artificially maintained distance between pro- 
grammers and managers, and Its negative effects on the production of computer 
systems. 

In addition to psychological factors that play a significant role hi the pro- 
gramming process, there are other, even more tangible pressures that affect the 
quality of computer programs, often by contributing to their inscrutability. The most 
obvious of these pressures are caused by short range economic constraints on aN 
programming projects. These constraints discourage the writing of thorough docu- 
mentation or of well commented, understandable code. It Isa common folklore 
among people who are not terribly familiar with computer programming that good 
code is likely to be more expensive to run than code that consists of program 
"hacks," 7 but tills notion is not supported by practical experience. What Is cer- 
tainly true is that obscure code is in the long run costlier to maintain than wefi writ- 
ten code. Unfortunately, the non-monetary costs of using poorly coded programs 
(which are bound to be unreliable and incomprehensible) are not reckoned into the 
budgets of the projects that produce them. 

Economic requirements translate into time ttmltations that tend to pressure 



6 Kraft, op. c/t. 



7 A vernacular term used to denote quick, clever pieces of work that are 
generally not very well structured or documented. 
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software engineers Into accepting Inadequate solutions to problems that they are 
asked to solve. Weinberg voices the opinion of everyone who has ever done any 
serious programming when he states that "we must be realistic and acknowledge 
that probably no perfect program was ever written. Every really large and 
significant program has Just one more bug." 8 Nevertheless, adequate time for care- 
ful program design, meticulous coding, extensive testing, and complete 
documentation - all of which are necessary If we are even to attempt to produce 
high quality, understandable code - Is considered a luxury m most non-academic 
programming situations. In general, It Is only the short terra achievement of a sys- 
tem that runs which is tangibly rewarded. Because the long term, global view Is 
not usually taken, Important considerations may be overlooked. For Instance, once 
a given computer system is declared operational and is pdt Into use, many of the 
programmers who were Involved In making that system run begin working on other 
systems; thus, the people who help design and Implement a large system are not 
usually around when problems with the system arise. 

Another factor that contributes to program incomprehensibility Is the gen- 
erally low level of professionalism that characterizes the activity of programming. 
We think of an activity as being professional when the members themselves deter- 
mine the criteria for membership, choose who is to be a member and who Is not, 
and have the power to remove someone from the profession If The does not adhere 
to certain collectively set technical and ethical standards. The Council of the 
Association for Computing Machinery has adopted a set of guidelines for profes- 
sional conduct in information processing, but computer programming Is not really 
governed by anything like a code of ethics; the fact Is that there is no 

8 Gerald Weinberg, The Psychology of Computer Programming (Mew York: 
Van Nostrand Reinhold Company, 1971), p. 10. 

Section 3.1: The Programming Process 4 "*- 



professional standing for programmers (the ACM primarily serves academics and 

g 

other computer science researchers;" that Is, people who make up a small portion 
of all people who work with computers). 

Current writings that deal with the training of programmers indicate that 
software workers are taught not to question the constraints that are imposed on 
their work, even though these constraints may be imposed by people who have had 
few dealings with computers; they are taught to stress corporate profitability and 
to identify this with technical rationality and efficiency, even though these criteria 
are not sufficient to determine when, a given performance level of a program is met 
and can be depended upon; in short, they are taught everything but a sense of 
professional ethics. Furthermore, they are subjected to continually changing 
specifications from computer users; these, combined with the coming and going of 
people involved in a computer project as the design of the system progresses, 
ensure steady modification that fundamentally changes a computer system in 
unpredictable ways. 

3.2: Complexity 

Some advocates of increasing the use of computers would have us believe 
that computers are both necessary and beneficial to a society as complicated as 
our own. They argue, on the one hand, that "without such Intellectual aids, our 
complex modern society may weH fall apart under Its own weight of complexity, " 
and, on the other hand, that better (/.e., computerized) means for handling and 



8 

This can easily be verified by a casual survey of the affiliations of the au- 

thors of the articles and letters in ACM pubiicatkirts. 

1 °Robert Fano, "The Computer Utility and the Community," IEEE International 
Convention Record, Part A 2, 32. 
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disseminating Information will result m an Increase In Intellectual freedom and In 
diversity of lifestyles. These arguments form the basis for much of present day 
computer use; yet, as I intend to show, they do not stand up to critical examina- 
tion. 

The value of the computer In dealing with really complex problems, and par- 
ticularly with human difficulties, Is highly questionable. In some cases, the complex- 
ity of computer applications derives not f row «•* problem domain, but from the use 
of computers. It Is actually the case that some computer systems generate com- 
plexity and render problems more Incomprehensible an* unmanageable than they 
were originally. For Instance, Edwin Paxson attempts to justify computerized mili- 
tary defense systems by stating that -there is little question that serious systems 
analysis without the computer Is Impossible"! He goes on to undermine hte own argu- 
ment by unknowingly naming the computer Itself as the source of the complexity 
with which it is associated: "As computing power has increased, so also has the 
complexity of the analyses It fosters." 1 ^ [emphasis mine] ShnHerty, John Kemeny 
first praises Dartmouth College's computerized bookkeeping system for maintaining 
many thousands of accounts accurately, and then acknowledges that "It is also a 

byproduct of the system that we have such a vast complexity of information . . . 

12 
that human beings cannot cope with it without the aid of the computer." in the 

light of what these people and others have said, the argument that computers are 

necessitated by the complexity of modern society Is hardly tenable. 

The alleged benefits of complex computerized information systems are even 

11 Edwin W. Paxson, "Computers and National Security," Chapter 3 in Com- 
puters and the Problems of Society, ed. by Harold Sackman and Harold Borko (New 
Jersey: AFIPS Press, 1 072), p. 77. 

12 John G. Kemeny, Man and the Computer (New York: Charles Scribner's 
Sons, 1972), p. 104. 
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easier to dismiss than the preceding "argument from necessity." According to one 

view, computers, by virtue of their Information processing capabilities, are part of 

the foundation of a new golden age. However, it is not difficult to see that existing 

computer systems do not provide any evidence of the coming of this prosperous 

future, and there is no compelling reason to believe that twice as many, or even 

ten times as many systems ever will. 

The record of how technology has actually been used in the 
modem world does not support [this] supposition^ in principle, 
transportation and communication might have been expected to 
contribute to a broadening of human experience and greater 
tolerance for different customs and beliefs .... The ability to 
travel with ease on a global scale ami communicate across 
great distances has not led to increased understanding or great 
compassion. T+iere is no evidence to suggest thai international 
tensions have been lowered as a consequence of increased 
information flow ... . 

The problems of human interaction cannot be reduced to 
Information flow, nor can the shortco mings of existing social 
arrangements be attributed to imperfections in our instruments 
of communication. Is there any reason to believe that # two- 
way terminal in the home would materially alter an Individual's 
response to television broadcasts showing scenes of violence 
or human mercy? The mere fact of having access to informa- 
tion does not create the disposition to act appropriately . ... 
It is purely wishful thinking to suppose that improved information 
flow win result in spontaneous efforts to resolve oonfict and 
create more responsive social environments. The historical evi- 
dence points to further concentration of power." * 

Mowshowitz drives home the point that although Information is necessary for 

rational decision making, it is not sufficient for harmonious social interaction. 

In discussing complexity, I do not mean to refer to the formal definitions of 

complexity utilized by computer science researchers, but rather to a more intuitive 

definition. A complex computer system is one that is composed of many parts that 

are interrelated in complicated ways, so that it may very well be impossible for us 



' 3 Abbe Mowshowitz, The Conquest of Will: Information Processing in Human 
Affairs (USA: Addlson-Wesley Publishing Company, 1970), pp. 164-65. 
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to keep track of all the systems' components and the relationships between them. 
In reference to truly complex computer applications, I have already discussed the 
Importance of a well understood theoretical foundation; a computer model (or any 
model, for that matter) Is only as good as the theory behind It. The point here Is 
that complexity is not a mandate for Incomprehensibility: witness the MACSYMA 
system (mentioned in Chapter 2) and the DENDRAL system, which Implements a 
theory of mass spectrometry. "... incomprehensibility Is not a necessary pro- 
perty of even huge computer systems. The secret of their comprehensiblllty lies in 

14 

that these systems are models of very robust theories. 11 

Most computer systems in use today "don't deal with complexity at all - nor 
are they designed to - they deal with sheer magnitude." 16 This is often the case 
with the huge Information banks utilized by so many businesses. In these applica- 
tions, computers have become necessary only In the sense that certain services 
could not be rendered In their present form without the computer. However, the 
present form of these functions has been dictated In large part by the availability 
of and early dependence on computers. This state of widespread computer use 
and dependence, which now appears inevitable and inescapable, is In reality man 
made. Apparent complexity may be deceiving and does not always warrant the use 
of computers, which may only conceal less technological, perhaps more perspica- 
cious ways of solving certain problems. 

I have mentioned complexity In my discussion of Incomprehensible computer 
systems because it Is one quality that Is nearly always associated with computers 



14 Joseph Welzenbaum, "Human Choice hi the Interstices of the 
Megamachine," p. 13- tectum |>re§§!?t*i «* &* .«:«?* Conference pif *Human 
Choice and Computers." Vienna, Austria, In Juna, 1979. 

16 /o/d., p. 7. 
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and that figures largely in their incomprehensibility. Shallow understanding of a 
complex problem may result in a system that is unnecessarily opaque. Unfor- 
tunately, our assumption that computer systems are too complex for the average 
person to understand has helped prevent the exposure of much existing system 
incomprehensibility. In dealing with computer systems, it is wed to consider both the 
easily forgotten fact that enormously complicated tasks were successfully carried 
out before the advent of computers, and the easily obscured fact that the modern 

4 fa 

megamachine has not heralded an age of human happiness. 

3.3: Evolutionary Systems 

One of the reasons that even computer scientists are worried about 
incomprehensible systems is the current reality of computer systems that have 
grown to a point where they are no longer under the control of the people who con- 
ceived and created them. A large system Is worked on by so many people over 
such a long period of time that by the time It is completed, there Is no person or 
small group of people who can be said to understand it. 

One of the most disturbing facts about large computer systems Is that there 
Is no group of people who can be identified as the authors of the system; in other 
words, no one is ultimately responsible for the operation of the system. Virtually all 
large computer systems are far too difficult to comprehend for anyone to be able to 
modify them without risking unpredictable consequences; certainly this is always 
the case with systems of whose evolution we cannot keep track. Once they are 
put into operation, these systems are, for the most part, immune to change - they 



10 A term used by Lewis Mumford to describe massive organizations Intended 
to carry out tasks whose magnitude place* ^Irfe^eyonlf the eapaWtrtiee of smaW 
groups of people. 
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can only grow. 

Herbert Simon, a leading computer specialist, describes a computer program 
as a strategy of action whose states and Inputs need not be envisioned In 
advance by the programmmer. 1 7 Such an attitude does not support the security 
derived from the commonly held view that computers do only what they are told to 
do. In a fearfully real sense, Simon is describing systems whose present form and 
mode of operation were neither planned not foreseen when the system was 
designed. These are systems that have evolved Into what they are now, In ways 
that no one has kept track of or really understands. 

When a program grows In power by an evolution of partially 
understood patches and fixe* the programmer begins to tose 
track of Internal details, loses Ms ability to predict what will 
happen, begins to hope Instead of know, and watches the 
results as though the program were an Individual whose range 
of behavior is uncertain; 

This is already true In some big programs ... it will soon 
be much more acute ... isfge heuristic- programs wilt be 
developed and modified by several programmers, each testing 
them on different examptee Trem dtWireft* eoneote* and inserting 
advice independently. The program wW grow hi effectiveness, 
but no one of the programmers wWu»a*rstai*d?1t siMOf course, 
this won't always mm&settf#- 1 *i*MeH*mm&n*ofc**a It 
worse, and no one might -be eWe iSf fttP* tgrtel) Now we see 
the real trouble with statements mte m enty does whet tts pro- 
grammer told It to do* There Isn't a*y one programmer. 

Professor Minsky might well have said that there are no indMdeel programmers at 
all. Gerald Weinberg has emphasized the team programming concept; 19 the sur- 
vival of the team throughout aH phases of a computer system^ from design through 

17 Herbert A. Simon, "What Computers Mean for Man and Society," Science, 
106 (March 18, 1977), 1187. 

18 Marvln Minsky, "Why Programming is a Good Medium for Expressing Poorly 
Understood and SteppMy Formulated Weaas*; J* ©es/fifr entf meaning, tt, ed. by M. 
Krampen and P. Seeltt {New York!* Hastings House, *©67); P- 121 — <*****, * 
Welzenbaum, Computer Poivet and HUtourt fleesom Proni Judgimot to CatculaUart, 
p. 236. 

19 Weinberg, op. clt. 
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maintenance, encourages the belief that there is a group of people who continually 
maintain control over the system. However, the composition of a team changes so 
often that the team that maintains a system and must deal with problems that arise 
in the course of using it is fundamentally different from the team that made the Ini- 
tial design decisions. Thus, in addition to the fact that large computer systems 
adapt as they operate, the group of people that is supposedly In control of the 
system continually changes. It may be impossible to reliably predict the final 
results of a computer system for quite some time after it has been put into opera- 
tion and after people have begun depending on it; it may also be Impossible to reli- 
ably predict how well a programming team understands a system at a given time. 

Much of the incomprehensibility tiiat characterizes these evolutionary com- 
puter systems derives from tiie organization of the programming of the 
system - the highly segmented, hierarchical structure of a large computer 
project - and from the programmers themselves - the constancy changing nature of 
the group of people who create and then maintain a computer system. 

Most of the routine and tedium associated wit|v Jtho daily tasks 
of producing programs are teft to an aflKwymou* a<*y of., people 
who merely do what they are told, uft d a rs t s n rtin fr ifttie of what 
they do and lee* of why they ero doing l** n :: A|}|fast MP until 
now, the computer has intensified, not reduced,, tite separation 
between those who think and those who do everything else, 
[my engine* m] 

Although such systems may not be smart enough for us to imagine depending 

on them a great deal, Marvin Minsky worries that "unfortunately, there are too 

many ways a dumb system with a huge data base can be useful." Just as one 



20 Kraft, p. 29. 



21 Marvin Mineky, "Computer Science and the Representation of Knowledge," 
hi The Computer Ag&t A Twentyr-Yoar Vtmm • od**<by Michael JL Dertpuzos and Joel 
Moses (Cambridge, MA: The MIT Press, Mi#fej^^ Caj&pe; in^iJosepb Welzenbaum, 
"Once More: The Computer Revolution," in Michael L. Dertouzos arid Joel Moses, ed., 
op. clt. 
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cannot talk usefully about a computer program without also talking about the tech- 
nological system in which It Is embedded, ene cannot talk usefully about a computer 
program without considering the database on which It draws for Information. It is 
often the case that one cannot distinguish very easily between the program and 
the database system. Modern databases are often dynamic, Introducing yet 
another source of unpredictability Into a computer system. 

It is not difficult to And people who seriously worry whether we can maintain 
control of our computer systems - whether people can keep up with their machines. 
It is harder, however, to And someone who recognizes the absurdity of needing to 

ask such questions. 

But we must win our technological race with competing nations 
first and then do the best we can with the realignment problem. 
Remember, the readjustment Piotoimm is common to all 
technologically-advancing nations. 

What Thompson euphemistically caMs the -realignment problem" Is really what 
Joseph Welzenbaum describes as -the feeling of powerlessness so ubiquitous 
among individuals m our society ... the widespread alienation of people from one 
another and from their work ... the perception of ordinary people that they are liv- 
ing In the Interstices of a gigantic system.- 23 Computer systems usually replace 
older, less technological ways of doing things, and once a workplace Is organized 
around computers, they quickly become Indlepensable to the functioning of that 
workplace. If no one is willing to assume responsibility for a system, why are we 
wining to trust that system In non-trivial areas of our Uvea? 



22 Howard Thompson, Joint Man/Machina Decisions (Cleveland, Ohio: Sys- 
tems Procedures Assoc., 1 066), p. 67. 

23 Weizenbaum, -Human Choice In the Interstices of the Megamachlne," p. 1. 
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Chapter 4: The Technological Society 

Of all the things Inappropriate to the man-made environment of 

the modern age, none i» ao Inappropriate as man himself. He 

must adapt himself.* EMul comments, 'as though the world were 

new, to a universe for which he was not created.* . 

' Langdon winner 

Quietly and complacently, It was sinking Into decadency, and 

progress had come to mean the prograaaof the Machine, 

E. M. Forster 

I have already noted that computer programs should not be considered 
independently of the circumstances In which they are used; similarly, computer sys- 
tems and system incomprehensibility should be examined from a broad perspective 
that emphasizes the technological society in which we live. Daniel Belt writes that 
"... technology is not a 'reified thing* or some abstract 'logical Imperative' but is 
embedded in a social support system, and it is the support system, not the technot- 

ogy, that determines its use." 

There are many aspects of society's attitude toward technology that can 
lead directly to incomprehensibility in computer systems. Our technological society 
operates m "performance mode." whereby i*sue* not dfcactly related to the 
economic value of a system are likely to be suppresseo\ Questions such as 
whether or not we need a particular system or whether ..§, system Is safe (in a 
broader sense than that exemplified by whether a system is harmless to Its 

1 Langdon Winner, Autonomous Technology {Cambridge, MA: The MIT Press, 
1977), p. 216. Quoting Jacques EHul, The Technological Society, trans, by John 
Wilkinson (New York: Alfred A. Knopf, 1904), p. S26V 

2 E. M. Forster, "The Machine Stops," In E. M. Forster, The E ««/» a/ / * fac/ » / " e 
an* Other Stories (New York: Harcourt Brace ^^J™^;*?^^ 
Abbe Mbwshowitz, The Conquest of Will: information fro&emng In Human Affairs 
(USA: Addlson-wesley Publishing Company, 1976), p. S§1& 

3 Daniel Bell, "Hard Questions and Soft Minds: A Reply to Wtfzenbaum," 
Chapter 21 in The Computer Age: A Twenty-Year View, ed. by Michael L. Dertouzos 
and Joel Moses (Cambridge, MA: The MIT Press, m press). 
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operators, for example) may be subdued by questions like "will it work?" 

There is widespread pressure against critics of computer systems, which 

encourages dependence on these systems. This often takes the form of peer 

pressure; counter critics cite the observable fact that "everyone else" seems to 

be using computers. 

And so we conclude that on-line decisions have to be made 
rapidly In some systems because of the advance In technology, 
and have to t»e mada raphfly in business to ifta^ timr competi- 
tion created by learning to make those vital, rapid decisions 
elsewhere. 

The above reasons - business competition and advances in technology - are typi- 
cal but not extremely compelling justifications for the often uncritical expansion of 
large scale computer applications and for the discouragement of criticism. 

4.1: Autonomous Technology 

The organization of modern society is founded on what many people perceive 

as technological necessity - a widespread sense of technological inevitability that 

Langdon Winner has called "autonomous technology": 

Autonomous technology Is the part of our being that has been 
transferred, transformed, and separated from living needs and 
creative intelligence. Any effort to reclaim this part of human 
life must at first seem mn^ctical sntf even absurd. 

Several writers have distinguished between technology and the broader connota- 
tions of technique - a limitless way of organteihg the world; an alHencompasstng 
arrangement of which human society is but one segment My intent throughout tills 
chapter is to examine the presence of technique in today's society. Most of us 



^Howard Thompson, Joint Man/ Machine Decisions (Cleveland, Ohio: Systems 
Procedures Assoc., 1066), p. 40. 



6 Wkmer, p. 333. 
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are governed to a larger extent than we consciously acknowledge by a technologi- 
cal mentality, according to which technique assumes the characteristics of an 
irresistible force and not of just a tool. Our sense of inner directedness has been 
superseded by a sense of what we believe our highly organized environment 
*wants" us to do or wHi allow us to do. 

The technological society Is, tea large extent, a dangerously uncritical one. 
We have allowed our virtues to become technical ones; the spirit of the day is that 
of maximum productivity. Technique "clarifies, arranges, and rationalizes; it does in 
the realm of the abstract what the machine did in the domatt* of labor." 6 Technique 
specifies attitudes that are valid once and for an. 

Winner exhorts us to consider "Instances in which things have become 
senselessly or Inappropriately efficient, speedy, rattonallzsd, measured, or techni- 
cally refined." 7 I have encountered numerous exampleiB d#-* *ough«ess accep- 
tance of technological virtues; t wfll mention only ^a*ew«f aiam* Howard Thompson 
(in Joint Man/Machine Decisions) implteitly assume* ^hotleeiraWHty of complexity 
and competitiveness; on the basis of this assumption, he does not bother to Justify 
the use of computer systems m^hw discussion of decision making. 8 The title of 
Sheridan and Ferrelf s book, Man-Machine Systems, tofoemmtlon, Control, ano* Deci- 
sion Models of Human Performance, would seem to indicate an emphasis on the 
human element Of man-machine Interactions. However, the authors Inform us that 
their consideration of human performance is necessarily limited to that which their 
models can describe: 



6 Jacques Ellul, The Technological Society (Hew York: Vintage Books, 1964), 
p. 6. 

7 Wlnner, p. 230. 

^Thompson, op. clt. 
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People may show grace, imagination, creativity, or feeling even 
In narrowly constrained tasks; but these queSties are too fine 
for the nets we cast in modeling and experiment. We have to 
be content to describe and predjot a* a mueh more mundane 
level. Our frequent use of terms such as operator and perfor- 
masce Ins4e«d of jDsrsoiirer ^ 

engineering context and the relatively narrow range of human 
experience which it enco mpas s es . --- 

Sheridan and Ferrell certainly deserve credit for recognizing and bringing to our 
attention some of the distinctions between the human, and the mechanical elements 
of the interactions that they consider. Still, when we are led to beUeve that we 
must be content with a narrow range of human experience because technique 
requires this limitation, tiien our lives are being viewed too much in the "engineer- 
ing context." "... to dwell on these impressive statistics which tell us what peo- 
ple oo, without attention to how they tee/ about what they do is to miss a pro- 
foundly Important: dimension of human experience; - that Is, the meaning that peo- 
ple attribute to their behavior." 1 9 

hi one article, Harvey Wheeler attempts to expiaia why we should use com- 
puters to aid us b* leaking decisions. He states that we do not expect perfection 
from either men or machines, but that aU we want is a way to make decisions more 
systematically. 11 Again, T cad attention to the unoAieationed assumptions - in this 
case, tits* systematielty as such simply and obvto us ^y Justifies itself and is always 
to be preferred over less formal criteria. Why sot go beyond the question of 
whether or not we can systematize the way ,te which we make decisions and ask 



°thomas Bi. Sheridan and William fr. Feitell, NtatrMachlne Systems: Informa- 
tion, Control, and Decision Models of Human Performance (Cambridge, MA: The MIT 
press, 1974). y - • 

10 Ullien BresJow Rubin A Worlds of Pain/ Ufa in the Working-Class Family 
(New Yoric: Basic Books, Inc., IfTSfc P^ f^S: c 

11 Harvey Wheeler, "Artificial Reasoning Machines and Society," Chapter 13 
In Computers and the Problems of Society, ed. by Harold Sackman and Harold Borko 
(new Jersey: AFIPS Press, 1972), p. 489. 
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whether or not we really want or ought to do so? 

The use of computers Is sometimes Justified by the argument that at least 
computers are no worse than people (e.g., automated decision making can hardly be 
worse than decision making by executives in a complex bureaucracy), and that the 
computer's rationality frequently makes them preferable to people In seme applica- 
tions. 12 Often, there is evidence of a pro-determined approval of toe use of com- 
puter technology that may cause us to ignore many undesirable aspects of applica- 
tions that computer systems efficiently cover up - aspects that cannot be 
effectively dealt with by a computer system. For example, toe ELIZA computer pro- 
gram (mentioned in Chapter 2) was hailed by some people as the precursor of a 
psychotherapy machine that would deal with much of toe neurotlclsm of modern 
society better than human therapists are able to do. People who made this Judg- 
ment tended to lose sight of the nature of human psychological problems amidst 
their raptures over the computer. "ELIZA had less to do with showing how much a 

computer can do than with revealing how coonttively and emotionally empty some 

1 3 
forms of human interaction can be . . . ." 

In our society, there is a strong temptation to submerge one's own autonomy 

in the megamachine; to give way to "a conscious and unconscious response to 

whatever situation arises. This response strongly and automatically repulses any 

alternative mode of action ... it neutralizes alternatives by making them seem 

unnatural, impractical, or simply impossible." 14 In minimizing the role of human 

12 Several writers have argued that, although computer systems may not 
perform as welt as superior human beings, the systems are better than most peo- 
ple. See Kenneth Colby, "Computer Psychotherapists," UCLA Department of Psychi- 
atry, Algorithmic Laboratory of Higher Mental Functions Memo ALHMF-14: 

13 Theodore Roszak, The Computer - A Little Lower Than the Angels," The 
Nation, 222 (May 1,1 976), 684. Review of Comput&r 'Power and Hum** Reasons 
From Judgment to Calculation, by Joseph Weizenbaum. 

1 dinner, p. 126. 
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beings in controlling their technical creations (be they computer systems or other 

technical systems), autonomous technology enables us to convince ourselves that 

the system progresses independently of our actions, and thereby to deny personal 

responsibility for our actions. 

it disturbs me to be told that technology "demands" an action 
the spealcer favo«^ that "yee cannot stop pflogress." It m>u> 
bles me that we are so easily pressured by purveyors of tech- 
nology mto permitting y :mm&m^.*p&9rmtm m bm- &m*--w m*s 
without attempting to control it - as if technology were an 
irrepressible force of nature to which we must meetdy submit. 
If we reflected, we might discover that not everything hailed as 
progress contributes to hsppirieest^t^ «» new is not ahways 
better nor the old always outdated. 

Finally, in a society in which technology has become an autonomous force, 

the only ruling principle appears to be that the technological system must be 

expanded, at whatever cost. Human agents are permitted to make decisions only 

according to criteria related to maximizing technological efficiency; but this is not 

real choice. 

To maximize energy, speed, or automation, without reference to 
the complex ccwHtiona that sMStah* organic fife, have become 
ends in themselves .... Under the pretext of saving labor, 
the ultimate end of this technics » to <keptaoe We, or father, ; to 
transfer the attributes of life to the machine and the mechant- 
cal collective, attewtrtg onry so muofr|ft tiui or^antsm to remain 
as may be controlled and manipulated. 

Mumford uses the phrase "the new Megatechnics" to describe the modern system 

whose chief purpose is control over the physical world, and ultimately over man 

himself. 



1 ^ice-Admiral H. G. Rickover, "A Humanistic Technology," In Technology and 
Society, ed. by Noel de ttevets <U8A: Addison^Ve«ley PubHsWng Company, 1872), 

p. 23. ^ ' - ■■-" - ; - ■■■' 

16 ElluK p 8a 

' 'Lewis Mumford, "Authoritarian and Democratic Technics," to Technology 
and Culture, ed. by Metvki Kraroberg and W Wlam H. Davenport (Mew York: 
Schocken Books, 1972), p. 66. 
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4.2: The Computer Revolution 

Computers are perhaps the most powerful playthings generated by modem 
technology. Unfortunately, computer systems are often used simply because they 
are there. We are too frequently guided by the technological maxim that any new 
technology that is possible must necessarily be utilized, whether or not It Is needed 
or desired. 

If a machine can yield a given result, it must be used to capa- 
city, and it is considered criminal and entisocW not to do *o. 
Technical automatism may not be Judged or questioned; immedi- 
ate use must be found for the most reoent, afeolent, and techn- 
ical process. 

Thus, Butler Lampson explains that we will see increasingly large computer pro- 
grams in the future, because of the availability of more and better computer 
hardware, 19 and Benjamin M. Rosen, an elestfoiteerae^psii^* analyst, says that 
Americans wiU indeed find a need for computers at home decease technological 
advances are being made so rapldty. 20 One artBbt ask why technotoglcal advances 
are being made so rapidly and discover that the rapidity of technological advance- 
ment is encouraged by the a p/7o/7 perception of »^s©g^ta>need for the fruits of 
this advancement; thus, th» minute size of this vicious clrcie is exposed. What is 
technically feasible is allowed to happen without regard to consequences. 

We have already seen that economic incentives and social fwressures com- 
bine to demand the use of computer systems. This situation has reached a point 
where any answer generated by a computer Is often^eeptable M us because of 

18 Ellul, p. 80. 

19 Butler Lampson, "Building Programs," MIT Laboratory for Computer Science 
Distinguished Lecturer Series, Msy 1,1^71.^ 

20 Mltchell Lynch, "A Computer Error: Trying to Use One ht Your Home," Watt 
Street Journal (May 14, 1979), 33. 
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its automatic component. The worst charge of a technological society is that of 

the impedance of technical automatism. In a very real sense, we cannot afford to 

doubt our computers. 

The so-called computer "revolution" consists largely of computer applications 

that are primarily guided by an implicit belief that computers should be used. This 

assumption is evident in many writings on the use of computers In psychotherapy, 

education, and military situations. Writers appear to start with the unjustified 

assumption that computers ought to be used to solve a given problem, and proceed 

to select for further consideration only those aspects of die problem that appear 

computable (refer to the following section for a discussion of how "the problem" is 

chosen In the first place). The problem itself almost appears to merit a status. 

secondary to that of the computer. 

Data models are tools. They do not contain In themselves 
the "true" structure of Information ... . He [a user] has to 
learn how to use rt We generally presume that this learning is 
required only because of the complexity *T the tool. DHUcurues 
are Initially perceived as a fatjure to fully understand the 
theory; there te an expectation that pers e v erance wM lead to a 
marvelous insight into how the theory fits the problem. In fact, 
much of We "learning" is reaBy a struggle to ecetrlwe some way 
of fitting his problem to the tool: changing toe way he thinks 
about Ms information, experimenting #Kh drfferent ways of 
representing it, and perhaps even abandoning some parts of his 
intended application because toe tool wont handle it Much of 
this "learning" process is really a conditioning of his percep- 
tions, so that he teams to accept as fact tho s e essumpttons 
needed to make the theory work, and to Ignore or reject as 
trMrt those cases where the theory faite. 21 

The question of whether a computer system Is the best solution, or even an 

appropriate solution, to a problem, and the question of whether we should direct our 

efforts elsewhere instead of forging ahead with another computer system, sre not 



21 William Kent, Data and Reality (New York: North-Holland Publishing Com- 
pany, 1078), pp. 184-106. 
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always asked when they should be. 

The usefulness of the computer is often assumed perhaps because of the 
awe with which our society generally regards science and technology. In our 
eagerness to find solutions to perpetual problems, we turn to what we believe we 
do best; in modern society, this is determined by the prodyctipn of solid results. 
"... in the West at least, the test is not so much what Mo you too*? or how 
elegant is your interpretation of wortdiy phenomena? but rather what can you actu- 
ally do?" 22 Thus, we assume that a theory of any Interpersonal activity can be 
expressed In the form of a computer program; that Improved means will triumph 
over carelessly considered ends; and that what appears to, the average person as 
the formal eloquence of the computer and the system behind it has the power to 
transmute errors into truths. 

It is in the realm of social difficulties that the computer revolution is particu- 
larly inappropriate. One computer scientist has described a not-too-distant future in 
which we will have access to a general purpose computing language that can 
describe any system that can be imagined. 23 In this way, we are encouraged to 
believe that a more fully computerized world wW eliminate existing social problems, 
"... to delude [ourselves] that gigantic Instruments can take the place of no 
ideas at all." 24 What we are not encouraged to do is to come to grips with the 
human sources of these problems, to consider how subjective, interpersonal factors 
may be computed, or to examine criteria according to which social problems may be 
considered "solved. " 



22 Winner, p. 25. 

23 Lampson, op. cit. 

24 Joseph Weizenbaum, "Human Choice in the Interstices of the 
Megamachine," p. 14. Lecture presented at the IFIPS Conference on "Human 
Choice and Computers," Vienna, Austria, in June, 1970. 
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Norbert Wiener points out some of the Important way* in which social activi- 
ties differ from the activities of computers: 



. . . learning machines must act according to some norm of good 
performance. In the case of game-playing machines, where the 
permissible moves are arbttfarBy est aU M alied In advance, and 
the object of the game is to win by a series of permissible 
rules according to a strict convention the* determines winning 
or losing, this norm creates no problem. However, there are 
many activreeS »at we should Wee to knprova by teeming 
processes In which the success of the activity is itself to be 
judged by a criterion Invohrtng human beings, and In which the 
problem of the reduction of this criterion to formal rules » far 
from easy. 26 [I would add that Such reduction is usually not 
possible at alt] 

To assign what purports to be precise values to such 
essentially vague quantities te neither useful nor honest, and 
any pretense of applying precise formulae to these Joosery 
defhed quantities 1e a sham and a waste iofttme.- , 'imy 
emphasis] 

Abbe Mowshowltz criticizes "automated common sense" - the substitution of formal 

processes for the intuitive decision making of an experienced manager. It is simply 

not always the case that all - or enough - of this intuitive knowledge can be made 

explicit; more often than not, valuable information is lost in the translation to the 

language of the computer. 



4.3: Defining the Problem 

A noticeable effect of the computer revolution is the frequent transformation 
of human difficulties Into a form that is amenable to a computerized remedy. By 
omitting the step of convincing ourselves that a given problem really is technologi- 
cally based, we are committing ourselves to forcing some problems Into a 



26 Worbert Wiener, Qod and Golem, Inc. (Cambridge, MA: The MIT Press, 
1064), pp. 78-77. 

26 /b/d., p. 91. 
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mold - that of a computer system - in which they may not fit. The nature of such 
problems can easily be subdued by the method of approach and the techniques 
employed. 

Many writers have remarked that the technological society is one in which 
our needs and desires are inevitably formulated as technological problems. It is 
the solution, e.g., the computer, that defines the problem. 

If the technique in question is not exactly adapted to a pro- 
posed human end. and if an individual pretends that h* is adapt 
ing the technique to the end, It Is generally quickly evident that 
it is the end which is being modified, not the technique. 

Writing about a more specific domain, P^.J^e,ft,g9pp«ji$»^JM,«l**ll^.to for " 
malLze the activity of programming by increasing the use of pre-tasted software 
routines: "in effect, the use of canned programs represents a joint decision by 
software sellers and software buyers to make the problems M Jtae solutions at 
hand." 28 Note Kraffs perception of the fept that the existing situation is the 
result of decisions that have JJ*en made by people. 

The area in which the computer shows up most glaringly as a solution looking 
for a problem^ that of ^eociafc problems, In wMchther* is most often no perceived 
need for computation, (for example, the "probjems^ that characterize our educa- 
tional or legal systems). Quick /iaeiinolegicajt tixes *qc social problems have a w«y 
of affecting and changing the proWems in u^ lM*t »> 

the application of patches to .computer programs compounds their 

27 Ettul, p. 141. 

28 »»itlp Kraft, Programmer* and Managers* lift* ftoutfnlzatloi* of Computer 
Programming in the United States (New York: Springer-Verlag, 1077), p. 35. Note 
that this difficulty is common to a« situations in which standardized- not 
personalized - solutions are applied to problems. I believe that.Kraft is primarily 
concerned with canned databases* management information fystems, etc., and not 
with relatively narmfess fattd gerrtHrAry mfpoftaht) ththj&s such as canned 
mathematical routines. 
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incomprehensibility, so the use of an entire computer system as a patch solution to 
a deep human difficulty can set in motion a dangerous propagation of incomprehensi- 
bility. 

In considering the application of computers to social problems, we should 
first contemplate our perception of human difficulties as "problems. 11 Joseph 
Welzenbaum has pointed out that problems such as a series of mathematical equa- 
tions have permanent solutions, but that this is not the case with interpersonal 
difficulties. These problems involve confBcts of Interest between people and can- 
not be understood solely in information processing terms; these problems cannot be 
understood without ffrst understanding peoptei Human problems (for instance, sys- 
tem Incomprehensibility) are not "solved" In the computattortaJ sense of the word; 
what would constitute a solution to a social problem? Rather, they are transformed 
into other problems that may be easier to live with than those they replace; 

We fall into a mode of problem solving when we feaHze, at some subcons- 
cious level, that in the realm of social difficulties; if!* subjective nature of problem 
definition renders problems hot jus* complex, birt extraordinarily dHftoult to deal 
with. Computer systems are comparatively easy & deal with. Thus, we concen- 
trate our energy on Improving military comimind and control svslems instead of 
questtontng the need for ever more complex end advancetl fonss of destrtJC«on; we 
extol th* virtues of computerized psycrmtherapy wrtheu^ exMmlnlng the reasons 
why Increasingly targe numbers of people seek psychotherapeutic help; and we try 
to introduce computers into schools on a massive scale without ever facing the 
reality of what is happening in American schools in ether than information theoreti- 
cal terms. We can convtnoe ourselves that computers ean solve our problems as 



Joseph Welzenbaum, "The Last Dream," The Conference Board Magazine, 
XIV, Mo. 7 (July, 1977), 41. 
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well as or better than people only by substituting mechanical gadgetry for human 
attention and by practicing interpersonal activities In inhumane ways. "The world 
becomes computerized when all human problems are reduced to technological prob- 



lems." 30 



4.4: The Reduction of Human Experience 

Perhaps the most regrettable effect of the technological structuring of 
.present day society is the reduction of people to Information processing organisms. 
Intelligence, once thought to be the exclusive domain of human beings, is now 
often defined operationally, so that we may speak of intelligent machines. At the 
1977 International Joint Conferences on Artificial Intelligence, Al researcher Edward 
Fredkin stated that the achievement of a "thinking machine" requires a combination 
of only engineering and science. 31 Simon and NewelTs General Problem Solver com- 
puter program is an attempt to Implement their belief that the elementary 
processes underlying human thinking are analogous to the information processing of 
a computer. 32 Dartmouth president John Kemeny sees no good reason for not 
assuming the intelligence of computers, because they manifest Intelligence in a 
scientifically testable (which is to say, extremely limited) sense. Kemeny equates 
apparent randomness (what we might call incomprehensibility) in a computer system 
with intuition and creativity in people, and impfleV^at randqm .computer behavior is 

33 
as desirable as creative human acts. 



30 Joseph Weizenbaum, "Apollo Agonistes," lecture presented at SUNY in 
Albany, ^lew York, April 20, 1879. 

31 Israel Shenker, "Man and Machine Match Minds at M.I.T.," New York 
Times, August 27, 1977, p. 8, quoting Fredkin. 

32 Herbert A. Simon, "What Computers Mean for Man and Society," Sctence, 
195 (March 18, 1977), 1186. 

33 John G. Kemeny, Man and the Computer (New York: Charles Server's 
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What I object to in such observations is their implicit reduction of people 
and of human experience; their identification of "the scientific conception of valid 
experience with the whole of existence." 34 Roszak notes that In equating people 
with machines, we can either raise machines up to our level or lower ourselves; he 
laments that we have done the latter - that we have reduced all human culture to 
the machine's limited capabilities. 36 Roszak is hardly alone in perceiving the gra- 
dual adaptation of human needs, desires, and thought processes according to the 
demands of technique; for instance, Winner writes of the shaping of human con- 
sciousness within narrow technical channels. 

In technological terms, an individual's social worth is proportional to his "pro- 
ductive capacity in a competitive labor market." 37 The production norms dictated 
by the goal of profitability often conflict with spontaneity and personal creativity. 
Individual participation in the technological society is tolerated only according to 
the degree of an individual's subordination to the search for efficiency; only that 
which is controllable is allowed to remain in man. Kemeny suggests that we may 
have to accommodate the computer systems that will play an increasingly major 
role in our lives; 33 whatever cannot be adapted will be eliminated. 

I have already mentioned that life in modern society is formulated as a suc- 



Sons, 1972), pp. 11, 18. 

34 Abbe Mowshowitz, The Conquest of Wills Information Processing In Human 
Affairs (USA: Addison-Wesley Publishing Company, 1976), p. 276. 

35 Roszak, p. 533. 

36 Winner, p. 127. 

3 Mowshowitz, p. 250. In reference to the emphasis on the economic value 
of people, consider the following: "But where human lives are at stake, and particu- 
larly when these people neve paid for their transportation, a considerably higher 
degree of safety Is required." Thompson, p. 87. 

3 Kemeny, op.clt. 
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cession of problems to be solved. Humans are viewed as problem solvers only, and 
are encouraged to resist sloppy human reason and intuition In favor of the artificial 
reasoning of machines. Maziish urges us to see man's nature as being continuous 
with his tools and machines. 89 People become Interchangeable as they grow 
Increasingly alienated from their work and from other people. Technological reduc- 
tionist!) can have nothing but an erosive effect on the self Image of people. 

It is not only the human being, but the entire human experience which is 
viewed from a reductionist viewpoint. Ethical and moral traditions that are not cost 
effective become obsolete; beauty becomes that which is well adapted to use. 
"The virtues of slow Information processing and labor done at a leisurely pace have 
long since been sacrificed to the norms of work appropriate to the electronic exem- 
plar. The idea that a task is something to be pondered or even savored Is entirely 
foreign to this mode of activrty. ,,w 

In order to accept a perception of life limited by technique, "humanity . . . 
has or will soon have transferred all Its attention to one aspect of Its being - it 
has sacrificed emotion for rational thought." 41 Thus, Herbert Simon identifies our 
most challenging problem, "the scientific problem of our age - how to understand 
ourselves more deeply." 42 In identifying interpersonal understanding as a scientific 
problem, Simon is severely limiting the role of human beings In the most human 
endeavor of all - understanding ourselves. As EHul wrote, "men do not need to 



39 Bruce Maziish, "The Fourth Discontinuity," in J. Mack Adams and Douglas 
H. Haden, Social Effects of Computer Use and Misuse (New York: John Wiley & 
Sons, 1976). 

40 Winner, p. 205. 

41 C. C. Gottlieb and A. Borodin, Soc/a/ Issues In Computing {Hew York: 
Academic Press, 1 973), p. 266. 

42 Shenker, quoting Simon. 
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understand each other in order to carry out the most important endeavors of our 
times." 43 



43 Ellul, p. 132. 
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Chapter 5: Results of Incomprshsnslbls Systems 

. . . technique, as a result of the perfection of means which it 
has placed at the disposal of modern man, has effectively 
suppressed the respite of time indispensable $o $Mt fhythm of 
life; between desire and the satisfaction of desire there is no 
longer the duration which is necessary for real choice and 
examination. There is no longer respite for reflecting or choos- 
ing or adapting oneself, or for acting ..op, wishing or putting one- 
self together. The rule of life is: No sooner said than done. 
Life has become a racecourse composed of instantaneous varia- 
tions of the universe, a succession of objective events which 
drag us along and lead us astray without anywhere affording us 
the .possibility of standing apart, taking stock, and ceasing to 

set. 

Jscques Ellul 

The use of computer systems, many of which are Incomprehensible to the 
people who work with them, is firmly established in today's society, so It is not 
surprising that these systems have had significant impacts on our society of users. 
I have already examined a variety of factors thst help explsin why, in many cases, 
these impacts are undesirable and unanticipated. For example, consider the follow- 
ing points: first, our motivations for using computer systems are often not related to 
the nature of the problem at hand, and second, our perception of problems Is fre- 
quently distorted by a pre-determined bias in favor of technological solutions. 

In the previous chapter, I discussed some characteristics of a technological 
society; in this chapter, I will narrow that discussion to a consideration of some of 
the effects of the widespread presence of large computer systems. Some of these 
effects have already been explored In my discussion of the sources of 
Incomprehensible systems; this is appropriate because technological systems often 
nurture attitudes and create conditions which support their continued use and 
expansion. Thus, certain issues I have previously examined - for instance, 



1 Jacques Ellul, The Technological Society (New York: Vintage Books, 1964), 
pp. 329-330. 
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autonomous technology, pro-technology biases, and the maintenance of what Is 
often only an illusion of complexity in computer systems - are factors which both 
lead to the generation and use of computer systems, e.rtd result from our continued 
utilization of and dependence on these systems, in a society dominated by a 
myriad of technologies, computers occupy a uniquely awe-inspiring position. This 
chapter is an examination of some drfftcult questions raised by our growing use of 
computer systems wfttert w* do net understand; among them, questions about 
dependence on automatfetrih/ generated output arttd responsibility for decisions 
made with the aid of computers. 

5.1: Rationality 

Scientific explanations derive both their power and their llmitedness from the 
method of abstraction and simplification by which science proceeds. The scientific 
method is tremendously useful, but in limited ways; only Information which is, In 
Kemen/s words, "scientifically testable," 2 can be utilized in the construction of 
scientific experiments, models, and theories. Such information constitutes only one 
small aspect - the scientifically quantifiable aspect - of the world. Real life situa- 
tions are characterized by an extreme richness of knowledge, much of which is 
"unscientific" and hence essentially unsuited for technical manipulation. Just as 
the reach for technological omnipotence continues to require the reduction of 
human beings to that which technology can explain and control, so the manifesta- 
tion of power that we attribute to computer technology has necessitated the 
reduction of problems to those with which a computer can deal. 

One key to the seemingly universal applicability of computer systems as 



2 John Kemeny, Man and the Computer (New York: Charles Scribner's Sons, 
1972), p. 11. 
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solutions to problems Is that before the application of computers, the problem 
domain is limited by being "rationalized." Problems must be made sufficiently expli- 
cit for Interpretation by a computer system; to computerize often means first to 
rationalize. In many cases, it Is this initial organizational effort, and not any com- 
puter system at all, that serves as the solution to the problem. Moreover, If the 
large expenditures of time, effort, and money needed to set up a large computer 
system result In a colossal failure, this failure is usually not linked to the question- 
able appropriateness of using computers in the first place; instead, a new problem 
is formed - that of the insufficiency of the logical composition of the problem area 
(this new "problem," hi turn, may be deemed suitable for computerization). For 
Instance, in writing of existing failures in the use of computers In conjunction with 
medical practice, Abbe Mowshowitz states that "the promise is enormous, but much 
depends on rationalizing the organization of health-care services." 8 The successful 
computerization of health care services depends more directly on the human effort 
of organizing the field than on the secondary step of bringing In computers. 

The dangers of excessive rationalization derive from the lack of considera- 
tion for values which do not seem quantifiable, and the consequent loss of Informa- 
tion in a purely rational planning or decision making process. The limiting rationality 
that computer systems demand encourages us to disregard the most difficult - that 
is, non-computable - aspects of a problem. Welzeebaum emphasizes the fact that 
computers process only information, not meanings. 4 Technique requires that certain 
aspects of problems be Ignored, and even more sigrrtfteantly^ determines which 

3 Mowshowitz, The Conquest of With information Proeesslng In Human Affairs 
(USA: Addlson-Wesley Publishing Company, 1976% p. 12$. 

*We1zehbawm, "Once More: The Computer Revolution,* In The Computer Age: 
A Twenty-Year View, ed. by Mlchaeft. Dertouzos an* Joel Moses (Cambridge, MA: 
The MIT Press, in press). 
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problems are to be seriously considered. "... we have permitted technological 

metaphors to so thoroughly invade our thought processes that we have finally abdh 

6 
cated to technology the very duty to formulate questions." 

5.2: Believing Computers 

Our Inability to comprehend many existing computer systems means that we 

must rely on the correctness of the systems; we have allowed such systems to 

become indispensable to us. 

We can't count on making such complex computations manually 
when we know that an enemy will use •computer. So we must 
rely on a computer for speed, which makes our decisions totally 
dependant upon both the availability and the accuracy of the 
computer when rapid decisions have to be made. 

The obvious risk is that of the increased impact of system errors as we increase 
our dependence on computer systems. 

Today, the population in general does not understand technological forces, 
but is kept submissive and content with the wide range of services offered. 
Different groups, such as business managers, must depend on the output of compu- 
terized information systems, although they do not have the time to supervise the 
collection of data or to satisfy themselves as to the reliability of their computer 
systems (nor are they encouraged to do so). 

. . . how utterly dependant we have become on our electronic 
super-tools, how essential we have permitted them to become, 
not that they were needed in the first place. 



^eizenbaum, "On the Impact of the Computer on Society," Science, 176 
(May 12, 1072), p. 622. 

°Howard Thompson, Joint Man/Machine Decisions (Cleveland, Ohio: Sys- 
tems Procedures Assoc., 1065), p. 40. 

7 Walzenbaum, "Human Choice In the Interstices of the Megamachjne.' Lec- 
ture presented at the tFIPS Conference on "Human Choice and Computers*? Vienna, 
Austria, in June, 1070. 
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The Issue of dependence takes on a special meaning In relation to 
Incomprehensibility. We have already seen that large computer systems are too 
complicated for anyone to directly monitor their operation. Thus, our day-to-day 
contact with computers must proceed largely on the basis of our belief in their 
correctness. When the computer systems we use are not understandable to USf 
this belief reduces to fatth in a technology that we have been taught is too compli- 
cated for anyone but a specialist to understand. We abdicate the responsibility for 
our decision making to a technology that frequently is not comprehensible, while at 
the same time attempting to maintain a feeling of control. Thus, we assure our- 
selves that "these versatile machines have become the galley slaves of capital- 
ism." 8 The real situation, however, is that our sense of control is largely illusory, 
and that we have been and continue to be largely unjustified in transferring respon- 
sibility to computer systems. This was made abundantly clear during the Vietnam 

era, when the Chairman of the Joint Chiefs of Staff regretted that "it is unfortunate 

g 
that we have become slaves to these damned computers." 

Many people are now making decisions to some extent on the basis of 
potentially unreliable computer generated output, and some of these decisions can 
have vital, nonreversible impacts. In some cases, "... computers can provide not 
only the Information on which decisions are made but can themselves make deci- 
sions." 10 At the very least, we must ask what kinds of decisions, if any, computers 
ought to make; we must decide whether the increased risk of error is worth the 
alleged gain in precision and rationality, if we depend more and more heavily on 



8 "The Computer Society," Time, Vol. 11, No. 8 (February 20, 1978), p. 50. 
9 lbid., p. 46, quoting Admiral Thomas Mower. 

10 Herbert Simon. What Computers Mean for Man and Society," Science, 195 
(March 18, 1977), p. 1187. 
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computers; and we must worry that the goals of a computer system on which we 
rely may not be our own goals: 

A goal-seeking mechanism will not necessarily seek our goals 
unless we design ft for that purpose, and In that designing we 
must foresee all steps of the process for which It is designed, 
instead of exercising a tentative fore«lgfrt ; wrm* goes^ to a 
certain point, and can be continued from that on as new 
difficulties arise. The penalties for errors of foresight, great as 
they are now, will be , enormously increased as automizatlon 
comes Into its full use. 

Lmgdon Winner warns of "the distinct possibility of going adrift In a vast sea of 

•19 . 

unintended consequences."** 



5.3: Technique and morality 13 

It is to a large extent the common perceptJon of computers, which may not 
have much relation to actual computer systems, tfiat has detefttftie^ the degree of 
our reliance on computer systems and the extent to which we have transferred 
human responsibility to automatic systems. The issue of responsibility can refer to 
different things in relation to computer systems; for Instance, the responsibfltty of 
people for the propagation of new systems. In considering tiie concept Of auto- 
nomous technology, we saw that often hew systems are constructed without expB- 
cit or conscious human approval. Mot only do we have trouble comprehending the 
systems we use, but it is often difficult for us to understand how and why they are 
created. 



11 Norbert Wiener, Goo* and Golem, Inc. (Cambridge, MA: The MIT Press, 
1877), p. 63. - 

12 Langdon Winner, Autonomous Technology (Cambridge, MA: The MIT Press, 
1977), p. 80. 

13 "Technlque never observes the distinction between moral and Immoral 
use. It tends, on the contrary, to create a con^tetery independent technical morali- 
ty." Ellul, p. 97. 
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Molra Is at work here - a fate that employs the free actjon of 
men to bring about ends that carry an aroma of necessity. 

The modern emphasis on a scientific analysis of behavior undermines personal 

autonomy by placing control of human actions in the environment Responsibility for 

the unbounded increase in our use of computer systems seems to fall only on a 

technology that is, or so it is claimed, value free. 

Technological elitism is at the root of much of the widespread avoidance of 
responsibility for Computer systems. We are taught that only the experts can 
know what is best for us; In Kenneth Laudon's words, thjere has t been "a denigra- 
tion of faith in the wisdom of ordinary cWzeiis^^.loqfaaae^^j^en^a^tton, which 
frequently results from the introduction of a computer system, formalizes and 
rigidities the prominence of those people who can claim, tQ understand computers. 
In Social Issues in Computing, Gottlieb and Borodin comment on the political power 
of technocrats: because politicians themselves, usually have no technical expertise, 
it is the technologists who define the alternatives for all of us . 

The people who appear to be most directly accountable for complex com- 
puter systems - computer scientists and the resea^ctiers who determine the state 
of the art - do not always manifest attitudes that are as cautious or as humble as 
their positions seem to dictate. Joshua Lederberg, for example, has said that there 
is no difference between the things computer should not do and the things people 
should not do; the only Important thing is to be sure the machines do not get out of 



1 dinner, p. 71. 

16 Kenneth C. Laudon, review of The Conquest of Willt iofptmaUon Process- 
Ing in Human Affairs, by Abbe Mowshowitz, in Science, 103 (September, 1976), 
p. 1111. 

16 C. C. Gottlieb and A. Borodin, Social Issues l» Competing CHew York: 
Academic Press, 1973), p. 223. 
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control. 1 7 It is not clear whose control Lederberg believes computers are presently 
subject to; many other people believe that computers have already grown beyond 
the limits of our control. Still others believe that some programs themselves are 
already exercising control: Herbert Simon says of automatic process control sys- 
tems that "their programs retain control over the ongoing process." When ques- 
tioned about how far artificial intelligence systems could go, Simon, with apparent 

disregard for Issues of responsibility, incomprehensibility, ratability, appropriateness 

19 
of use, etc., replied only that "we'll know that when we're done." 

Since, as we have seen, a large system has no Identifiable group of authors, 

there is usually no one who feels directly responsible for the output of the systems 

and for decisions which make use of that output Currently, accountability for the 

reliability of computer systems is so vaguely defined and so well "distributed" that 

it is fundamentally nonexistent. This is so despite the fact that many people are 

concerned about our tendency to allow computer systems to become ultimate 

authorities which require little Justification. 

If the activities carried out by computers cannot be readily 
monitored and guided by people, and If human processing of 
information cannot be easily Intermixed with computer process- 
ing, computers tend to become unchallengeable author- 
ities ... 

Recall the discussion of theory of behavior in Chapter 2, where I noted that 

comprehensible systems are likely to be founded on well understood theoretical 



17 Lee Dembart, "Experts Argue Whether Computers Could Reason, and If 
They Should," New York Times, May 8, 1077, p. 34. 

18 Herbert Simon, "What Computers Mean for Man and Society," Science, 
196 (March 18, 1977), p. 1187. 

19 /b/d. 

20 Robert Fano, "On the Social Rote of Computer Communications," Proceed- 
ings of the IEEE, 60, No. 1 1 (November, 1972), p. 1261. 
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bases, so that they serve as models of a theory and not as unquestionable authori- 
ties. In the case of incomprehensible systems, we have noted that there is fre- 
quently no well based theory of use; the system itself is the theory of its use. 
Thus, whoever doubts the system finds himself in conflict not with a theory but with 
an enormous, incomprehensible programming patchwork. Nevertheless, some people 
have gone to the extreme of advocating that computers be held responsible for 
themselves; Howard Thompson believes in letting the machine be responsible for its 
share of the decision making load in Joint man/machine decisions. 21 The question I 
must ask is what meaning could machine responsibility possibly have in a human 

world? 

The habit of speech, and it surely reflects a habit of thought, 
that makes instruments responsible for events, leads directly to 
speaking and thinking of science and technology as autonomous 
forces and to the idea of technological Inevitability. It leads 
finally to the proposition that man Is, after all, Impotent to strug- 
gle with powerful impersonal agencies of his own making over 



which he has tost control, and that he Is therefore Justified 
abdicating responsibility for the consequences of his acts. 



In 



21 Thompson, pp. 27-28. 

22 Joseph Weizenbaum, "Controversies and Responsibilities," Datamation (No- 
vember 15, 1979), p. 173. 
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Chapter 6: What to Do About Incomprehensible Systems 

We must return to the human center. We must challenge this 

authoritarian system that has given to an underdlmensloned 

ideology and technology the authority that beton^gs to the human 

personality. I repeat: life cannot be delegated. 

Lewis Mumford 

I would like to end my discussion of incomprehensible computer systems on a 
relatively positive note; in this final chapter, I consider suggested means of dealing 
with the existence of incomprehensible systems. The chapter begins with a dis- 
cussion of program verification techniques that is more technical than that encoun- 
tered in the rest of this thesis. Even in this area of formal study, many of the 
difficult problems are not technical ones and cannot be solved solely by studying 
computer programs: for example, the problem of how to specify what a program is 
supposed to do. This chapter, IHce previous ones, expands from a program-oriented 
viewpoint (which in this instance verification studies exemplify) to one concerned 
primarily with systems (both technical and social). 

It should be noted that the following discussion pertains even to comprehen- 
sible computer systems; in fact, to all modern technological systems. Examination 
of well understood systems wHI generally reveal that many of the programming 
techniques described below were used, perhaps In modified forms, in the construc- 
tion of the systems. However, evidence of the ethical "techniques" which are dis- 
cussed In the latter part of this chapter (criticism of technology, acceptance of 
individual responsibility, and humanizatton of technological systems) Is harder to 
find, even In systems that we might not label technically incomprehensible. Deal- 
ings with systems that are incomprehensible do raise unique ethical problems, 

y Lewis Mumford, "Authoritarian and Democratic Technics," In Technology and 
Culture, ed. by Melvin Kranzberg and William H. Davenport (New York: Schocken 
Books, 1 972), p. 58 
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because it is In these Interactions that the necessity of trusting computers is impli- 
cit, and our vulnerability is greatest. 

We already know that incomprehensibility in computer systems can be mani- 
fested in many ways. A social form of technological incomprehensibility is reflected 
by the pervasive sense that most people have of not occupying a meaningful posi- 
tion hi modem technological society. For the most part, we do not understand how 
our technological systems operate - we do not know how to crittelzef technology, 
how to judge the extent to which we should depend on It, etc. It Is especially 
because of these social effects of Incomprehensibility that we feel no control over 
the use and expansion of computer technology and are unwHiing to assume respon- 
sibility for It; it is these issues that are addressed in the second half of this 
chapter. 

6.1: Program Verification 

Verification has proved to be a difficult term to satisfactorily define, largely 
because of the human factors Involved, first in the creation of the programs which 
are to be verified (programs carry with tfiem their programmers' Intentions, which 
are often unclear), and tlten In the mterprettrtten of the proof of correctness (we 
want to be able to 'trusf programs, out we each have different criteria for believ- 
ing a correctness proof). One suggested definition is the following: program 
verification is that- -branch of computer science whose goal Is to establish "whether 
a [given] program performs Its Intended task.* 2 What remain unanswered are the 
questions of what does It mean to talk about the intentions of a programming task, 



2 Barbara H. Liskov and Valdls Berzins, "An Appraisal of Program 
Specifications," MIT Laboratory for Computer Science CS© Memo 14T (July, 1076), 
P. 2. 
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and what does the assurance offered by a verification proof buy us? 

The need for a discipline like program verification springs from two conditions 
that have existed for some time: (1) professional dislike for flaws In finished pro- 
ducts (In this case, computer programs), and (2) increased dependence on pro- 
grams that can have extremely destructive effects. On the one hand, there Is a 
desire for certainty that has led us to mathematical formalism for a certification of 
program correctness. On the other hand (and more importantly), now that we have 
created computer systems that affect vital areas of our lives, we are beginning to 
wonder how we can depend upon the Information that we get from computers. 
Because "even minor errors [let alone grossly misconceived "designs"] can have 
serious consequences and be costly to fix," 3 the rote of verification In increasing 
our trust In computer systems is both methodologically and ethically Important, in 
terms of both physical and social costs, we cannot afford to trust unreliable com- 
puter software. 

It is important that the reader recognize the Inherent limitations of the 
verification approach to computer systems, before launching into the following dis- 
cussion. Verification studies are directed toward computer programs. Verification 
researchers are committed to eliminating the relatively low level, technical form of 
incomprehensibility that is characterized by program errors. Recall from the first 
pages of this thesis that it is the incomprehensibility of systems with which I am 
primarily concerned; this is a much more subtle difficulty than that addressed by 
work in the field of program verification (in fact, as 1 have tried to explain, it is not 
a technical issue at all). Although there is a significant role that verification proofs 



3 Susan L. Gerhart and Lawrence Yelowitz, "Observations of Fallibility in Ap- 
plications of Modern Programming Methodologies," IEEE Transactions on Software En- 
gineering, Vol. SE-2, No. 3 (September, 1976), p. 195. 
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can fulfill, extrapolation from a proof of correctness to a meaningful statement 
about the comprehenslbility of a system (and hence about the appropriateness of 
trusting that system) is, at best, unjustified. 

6.1.1: Automatic Verification 

The most widespread basis for verifying programs is the method of intermedi- 
ate assertions. Tags are placed at key points throughout a program (e.g., loops), 
indicating the state that the program Is supposed to be in whenever it passes 
each point at execution time. The central Idea is that aH loops in a program must 
be "broken"; /.e., it must not be possible to do a loop iteration without going 
through a tag. The intermediate assertions may relate values of program variables 
at intermediate points to initial values or to ultimate values. The program 
specifications for the initial state form the first assertion, and the desired output 
conditions form the final assertion. A proof of program correctness is divided into a 
number of smaller proofs that a program coming from assertion n wNI always satisfy 
the conditions of assertion n+1. The combination of these Intermediate proofs 
establishes the partial correctness of the entire program. 

Automatic methods of program verification center around the mechanization 

of the above approach (which might be called the informal assertion approach) 

through the use of automatic theorem-proving programs. 

The inductive assertion method reduces program correctness to 
a flrUt* set of finite patfts. A program p*itt start* wtth an Initial 
assertion, continues with executable code, and terminates with 
a final assertion. For each progra* path there^fii a togicaf for- 
mula called a verification condition. 

The first quest is for a system that would automatically generate the verification 



*Ben Wegbreit, "Constructive Methoda i in Program Verification," Xerox Palo 
Alto Research Center (December, 1 976), p. 6. 
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conditions. Once that Is done, the proof of the correctness of a program reduces 
to a proof of a theorem In the first-order predicate calculus (higher-order systems 
are also being examined, but less effort Is being devoted to them). The theorem 
establishes the partial correctness of a program, so a separate termination proof is 
required (in most discussions of program verification, it Is deemed easier to divide a 
total correctness proof into two proofs, one of partial correctness - or correctness 
assuming termination - and one of termination). According to this view, a 
verification system consists of a verification condition generator and a theorem 
prover. In practice, the step of proving that the verification conditions are true 
has constituted the bulk of work in this area. The preliminary step of finding 
appropriate intermediate assertions is still too little understood to be automated. 

There are a variety of problems, both technical and social, facing research- 
ers in the field of automatic program verification. Some of the technical difficulties 
may only require further study to be overcome, but some of the constraints 
imposed by less formal problems may represent Inherent obstacles to the success 
of verification proofs in increasing our trust in computer programs. Significant 
difficulties that are currently being addressed include the foBowing: 

• It is the hope of verification researchers that the verification of a compu- 
tation is much easier than the original computation. However, time constraints place 
a heavy burden on any verification system. Although a given system may be able 
to verify a large class of programs, "we are more interested In what the theorem- 
prover can do in 'reasonable' time." 6 Thus, even a complete verification system 
would not necessarily satisfy the practical time limits that it would have to be sub- 
ject to. Contemporary systems tend to flounder on complex programs which are 

6 Bernard Elspas, et. a/., "An Assessment of Techniques for Proving Programs 
Correct," Computing Surveys, Vol. 4, No. 2 (June, 1 972), p. 1 27. 
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founded on deep theorems, because the search space is increased to a point 
where a proof cannot be generated in a tolerable length of time (and, even when it 
is generated, it is prohibitively long and complicated). "We simply do not yet see 
how to prove programs are correct In any reasonably short manner at the present 
time." 6 



• Until now, verification proofs have dealt with relatively simple cases (highly 
restricted programming languages and ideal machines), and are not yet up to the 
level of complicated programs, where they would be useful. Some relevant techni- 
cal issues that are acknowledged as problems but have not been satisfactorily 
solved yet are indeterminacy, parallelism, exception errors (e.g., overflow and 
underflow), and side-effects of a given programming language. 

In addition, there is the difficulty of run-time errors and of specifying "the 
behavior of a program when an error Is detected during execution." Researchers 
in program reliability attempt to anticipate a range of possible execution errors 
ahead of execution time, so that error-handling measures can be Included in the ori- 
ginal design of the program. According to the philosophy underlying reliability work, 
errors are not necessarily eliminated, but they are anticipated and dealt with in 
understandable, acceptable, reliable ways. 

• Some computer scientists are concerned about the believability of proofs, 
a criterion that would not be well satisfied by a system that receives a program 
whose correctness it is to establish as input and outputs a huge proof that is even 
less comprehensible than was the original program. A verification system's 



6 James Joyce, "Human Factors in Software Engineering," in The First West 
Coast Computer >£*fr^^ Jr. (Palo 

Alto, CA: Computer Faire, 1977), p. 61. 

Uskov and Berzins, p. 18. 
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response of "QED M is not very meaningful If programmers have no reason to trust 
the system. One primary function of a proof of correctness Is "to dramatically 
Increase one's confidence In the correct functioning of a particular piece of 
software," 8 but before this can be accomplished, confidence in the proof process 
itself must be established. At worst, a complicated verification system could con- 
stitute an additional layer of Incomprehensibility between a computer program and 

its users. 

DeMillo, et. a/., convincingly argue that mathematical proofs come to be 
believed because of the existence of a social network In which proofs are widely 
read, ref ereed, published, reviewed, discussed, and finally, Internalized, para- 
phrased, and used. 9 Although the motivations behind mathematical proofs and pro- 
gram verification proofs are different, the concept of believablllty Is related similarly 
to both. Even a "correct" proof will not be used If It is not believed (belief here is 
defined in the social sense that DeMillo, et. a/., discuss), and DeMillo, et. a/, do not 
believe that a social process in computer science analogous to that in mathematics 
Is yet well developed. They also remind us that "the decision to consider a "proof 
in detail is often influenced by some slightly Irrational concern - 'how does the 
problem fee/r" 10 and urge us to strive for the simplicity that characterizes the 
most Important mathematical theorems and proofs. Finally, since any proof, no 
matter how formal, can be interpreted In different ways by different people, we 
should be aware that, In a practical sense, "a derivation of a theorem or a 



8 Richard A. DeMillo, Richard S. Upton, and Alan J. Per«*V "Social Processes 
and Proofs of Theorems and Programs," Yale UnlyersKv of Qomputerr Science 
Research Report #82 (1076), p. 1. 

°lbld., pp. 6-9. 

^°lbld., p. 10. 
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verification of a proof [can have] only probabilistic validity .'" The classical view 
of judging things in a strictly dualistic fashion is not applicable to verification 
proofs of computer programs. We must be wise enough to recognize the margin of 
error in any assessment of program correctness, and to require "another view of 

12 

'reliable design' . . . that more fully exploits the social mechanisms" to comple- 
ment the view taken by verification studies. 

• Computer systems are dynamic entities that may be vaguely specified. 
The foundations of modern "software systems are large programs with 
specifications and related documentation much larger than their code. More impor- 
tantly, when specifying a system it is often impossible to state precisely what is to 
be done. Typically some claims are made about what must happen and others 
describe desirable but less crucial behavior." 13 In practice, specifications change 
and grow as a programming project progresses, reflecting a restructuring of the ori- 
ginal purpose of the system and the original perception of the problem domain. The 
evolution of a system to meet the new criteria is usually not wed controlled. "The 
incompleteness and imprecision of the specifications for systems makes rigorous 
verification difficult and the impermanence of the specifications reduces the 
rewards of producing such a verification." 

e The interface between an automatic verification system and a programmer 
must remain informal. What cannot be completely formalized in this interface Is the 
purpose of the program, an informal, often unstated criterion. "... it becomes 



1 1 Gerhart and Yatowltz, p. 206. 
12 /W(f.,f. 197. 

13 Charies Rich, Howard E. Shrobe, and Richard C. Waters, "Computer Aided 
Evolutionary Design for Software Engineering," M.I.T. Artificial intelligence Laborato- 
ry A. I. Memo 606 (January, 1079), p. 3. 
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possible to formally prove consistency of programs with . . . formal specifications. 

However, the complimentary step of verifying that a program specification imple- 

ments the underlying concept must necessarily remain informal." We are faced 

with the problem of whether or not our specifications are strong enough to express 

our intentions. The most responsible attitude we can take is that "we can never be 

sure that the specifications are correct." 18 With such an attitude, we cannot be 

certain that a program that has been verified to be correct (assuming we achieve 

such results some day) will do what we want it to do, unless we are sure that our 

intentions have been accurately and completely codified in the program 

specifications. 

In verifying a program, the system assures us that the program 
satisfies the specifications we have provided, it cannot deter- 
mine, however, whether those specifications accurately reflect 
the intentions of the programmer. The Jpteni^s. a«»r all, exist 
only in the mind of the programmer, and are inaccessible to a 
program verification system. If he has made an error in 
expressing them, the system has no way of detecting the 
discrepancy. 

Some of the most promising current research in verification deals with ways in 

1 8 

which a system could detect the kind of discrepancy mentioned above. However, 
at least for the time being and particularly In the case of programs with vital 
consequences, an awareness of this 'discrepancy* should play a crucial role in our 



16 Zohar Marina and Richard Waldlnger, "An Appraisal of Program 
Specifications," Stanford Al Lab memo AIM-298 (August, 1 9 77 )» P- 24. 

l6 /o/d. 

1 7 Gerhart and Yelowitz, p. 205. 

18 For example, the notion of a "programmer's apprentice," which Is "a com- 
puter aided design tool which can help a programmer deal with program evolution 
from the initial design phase right through the continuing maintenance phase." 
Here, the effort is to provide "support during the process of developing code good 
enough to warrant the effort of certification." See Rich, Shrobe, and Waters, "Com- 
puter Aided Evolutionary Design for Software Engineering" (quotations are from 
pp. 1-2). 
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decision to depend on a particular computer program. 

6.1.2: Informal .Verification 

What is urged throughout the field of program verification Is the realization 
that "formalism should supplement, definitely not replace, common sense and pro- 

1Q 

gramming experience." Traditional verifying methods, which may all reduce to 
good programming standards, should be retained, along with a healthy skepticism 
towards formal proofs of correctness. Intuition or practical Judgment can detect 
many programming errors. One of the most elementary insights that comes from a 
study of verification is that programming is a human activity and that there are 
parts of the programming process that are best handled with some measure of 
informality. ■■'•'"- i;;,; "^ ■■' : -^r-~- i *< ^- 

In the realm of informal, manual program vertltcatldh, Hie most fundamental 
consideration IS a statement of "what should be proved in ottter to guarantee that 
program is correct . . . ."*" Since program specifications are generally the first con- 
tact one has with the ideas that will eventually be embodied in the program, Uskov 
and Berzins have emphasized the vahie of formal specHteaflona. Their argument Is 
that the increased rigor of a formal spectflcaften tacHftates agreement among pro- 
grammers on the meaning of the apectftoatton and, uH W ia l uly , the meanfng of the 
program. The likelihood of such agreement is increased by the formal nature of the 
specifications, which should help limit the number and scope of possible interpreta- 
tions. 

More specifically* the precisely-defined syntax and semantics of a formal 



18 Gefhart and Yetowftz, p. 206. 
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specification language should facilitate the construction of partial correctness 
proofs for the individual modules Into which a program can be composed. This 
proof, combined with requirements for module termination, would constitute a 
verification of the program. Liskov and Berzins conclude their paper by expressing 
the current need for proof techniques for the various specification methods. 

There are undeniable benefits from the use of formal program Specifications; 
for example, the ability to decompose proofs of program properties, and the ten- 
dency of a formal statement to bring out details that, In an Informal specification, 
might easily remain incompletely thought out and hidden under vague descriptions. 
It is important to note that increased formality In program specifications can 
decrease, but not eliminate, the inherent ambiguity Of the early stage of develop- 
ment of a programming project. The informal nature of most existing speclf&atiens 
does succeed in presenting the main points in a fashion that is more understand- 
able to most programmers than formal mathematical Statements. Liskov and 
Berzins recognize the role of informal specifications as a valuable and necessary 
complement to formal specifications. Their ideas cart be taken as suggestions for 
improvement in the practice of software engineering. It is clear that a more 
responsible attitude towards program specifications should be nurtured, so as to 
eliminate a variety of bad programming habits (such as "the common habit of writing 
the specifications after writing the program 1122 ) and the potentially dangerous 
effects they can lead to. Imprecise, loosely conceived design criteria are not likely 
to support comprehensible systems. 



21 Moreover - and this point cannot be overemphasized - most of- the pro- 
grams that I have been discussing ail along are in a domain where formal 
specifications are Impossible. In ^ramstcier^ §h* «l^ appli- 

cations, one must question how many offti*ajrafe fotmaHiabJe* = ?? 

22 Li8kov and Berzins, p. 3. 
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The method of intermediate assertions, ajready discussed, is useful in break- 
ing up the verification process Into manageable units, and in alerting the program- 
mer to particular areas of error (for example, the error must be In the code which 
lies between assertions 5 and 6). In addition, If the assertions are defined before 
coding, increased modularity is imposed; the goal Is to divide a large system into a 
number of smaller, more manageable, and hopefully .more easily comprehensible 
units. Elspas, et. a/., suggest "creating the assertions prior to formulating the 
details of the program algorithm, proving that the assertions correctly express the 
intent of the program, and, finally, writing the code that lies between the asser- 
tions." 23 

DA 

The difficulty of precisely defining the intent of a program notwithstanding, 
the Intermediate assertion approach seems to hold great promise for successful 
program verification. Jt has already affected the way we write software in a posi- 
tive manner, by providing another method of evaluating the meaning of a program or 
program segment. "There is no doubt that many programs could benefit from an 
attempt to carry out an Informal proof, at least for the fact that such a proof would 
reveal gross misunderstandings in the intended algorithm."^" 

An extension of the method of intermediate assertions Is Wegbreffs scheme 
of program Justifications, A justification is both a program assertion and a state- 
ment of how the path it applies to is to be proven correct. Justifications help clar- 
ify correctness proofs, but their real value Is that they serve as additional text of 
the program itself, along with code and correctness specifications. In effect, the 



23 Elapas, et, a/., p. 142. 

^*l do not mean to dispense Hghtiy with cttfflcuWes Wee tiila one, as I beHeve 
that they ultimately determine the extreme Mnitednese of pMOof* of correctness. 

Z6 Elspas, et. a/., p. 1 1 9. 
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programmer who utilizes this method proves the program correct while coding It. 

Wegbreit's common sense Ideas are instructive both in the study of 
verification and in the fulfillment of present programming tasks. In particular, he 
suggests "shifting part of the activity of program verification to language design 
. . . and to programming practice" 2 * and contends that "program correctness Is 
best achieved by explicitly considering the proof as part of the programming pro- 
cess." 27 Wegbrelt calls for a change in the way programmers think about their 
work - an increased awareness that reliability Is the responsibility of the program- 
mer. 

Other researchers have cautioned us not to abandon more "mundane" 

methods of program verification in favor of newly developed strategies that carry 

with them the legitimacy of mathematical formalism. The complex methods that 

researchers may find interesting are not always practical for programmers' uses. 

The kinds of algorithms that get 'proved* correct have nothing 
to do with software; given a choice J*etwe#n a yejy t good algo- 
rithm with a proof of correctness, but which may be hard to 
understand, and a straightforward, unproved algorithm whitman 
implementer believes he understands, the complex algorithm 
invariably loses,. Jlnd it is the complex algorithms that are raoet 
interesting and have the most chance of being subjected to the 
, sociology of 'proof.'" 28 

In a similar spirit, Gerhart and Yelowitz remark that a common feature of program 

errors seems to be a "tendency to concentrate more effort on the harder parts 

which require sophisticated techniques and less effort ,,0*1 the 'obvious' and easier 

parts," 2 - Sophistication should not overshadow tboroughnass. 



26 Wegbreit, p. 31. 



27 lbld. 



28 DeMillo, Upton, and Perils, p. 13. 
29 Gerhart and Yelowitz, p. 205. 
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A variety of good programming practices have evolved that in effect consti- 
tute informal efforts to verify software. For Instance, through the years, there has 
been an increasing emphasis on debugging; now, well developed debugging tools 
are commonplace In most large programming projects. It is crucial to note that 
verification methods that have evolved from the practical experiences of program- 
mers Involve, first and foremost, an expectation of program errors. 

it is clear that some common programming habits must be overcome if pro- 
gram verification techniques are to take hold, and that considerable effort will be 
required to do this successfully. "The methodologies proposed to increase 
software reliability are still in their early stages of development. The tasks are not 
easily taught or learned ... ." 30 For instance, it is all too often the case that 
specifications and documentation are not regarded as Integral parts of a program- 
ming project; yet, the effort that goes Into these attempts to clearly state what a 
program is supposed to do is essential to the success of any verification process. 
Without this thoughtful planning and documentation of programming, reliable large 
scale systems are hot possible. Until tf» mystique is removed from many people's 
concepts of programming and untH programmers come to recognize the ethical (and 
not Just the economic) value of dependable software, program verification will not 
be abfe to achieve its full potential. 

6. 1.8: Conclusion* About VoilW ue U oh ' 

Some common sense conclusions can be drawn from the preceding discussion 
of program verification. The first is ready a lesson In humility. "We can wan t 
correctness in our programs, but we must settle for reliability. " ° Abstract, ideal 



31 /b/d, p. 206. 
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notions of correctness In terms of perfection are not likely to be fulfilled by real 
computer programs; this must be acknowledged In the course of making decisions 
about the use of "verified" programs. "We simply must learn to live with fafflblF 
Ity," 32 and we must learn to do so responsibly, by not treating verified programs as 
perfect programs. ". . . In mature engineering disciplines, VeWao/e' never means 
'perfect . . . engineers set probable limits of failure, relying on other design criteria 
to place these limits well above the conditions likely to be encountered In prac- 
tice." 83 Current verification methods are best exercised with caution; that Is, with 
an understanding that although they are helpful debugging tools, they do not 
guarantee correctness. 

The second conclusion is that given a choice between Informal and automatic 
methods of verification, moderation Is most appropriate. "Experience with both . . . 
should convince us that neither type of evidence Is sufficient and that both types 
are necessary." 34 Verification studies have much to offer In increasing our 
confidence In computer programs, but confidence is built up on many levels, and 
different kinds of verifteation evidence are needed. We come to believe things for 
many reasons; formally structured proofs, Informal, Intuitive explanations, trial and 
error, and Insight all play Important roles in Inspiring trust In software. The best 
verification "package" will appeal to as many different channels of knowledge as 

possible. 

Thirdly, computer scientists should begin to place more emphasis on the ethi- 
cal issues that the use of any vital computer system brings into play. Software 
reliability is typically measured by the number of "bugs* encountered In a period of 

32 DeMlllo, Lipton, and Perlis, pp. 16-17. 

33 Joyce, p. 61. 

34 /©/d. 
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time ("mean time between failures" or "mean time to recover"). We must decide in 
what situations these standards of reliability, and the systems that cannot be 
trusted beyond these limits, are acceptable. At times, we may want to question 
the basic criteria for the use of computers in a given application; the degree of 
vitality of the effects of a system, the reversibility of those effects, and the 
comprehenalbillty of the system are some Issues that should be considered. At the 
very least, the dangers of overdependance on computer systems must be avoided 
by careful consideration of the ramifications of a dependant relationship between 
man and machine. 

Lastly, I would go back to my first words about verification, at the beginning 
of this section, and, state again that the goals of the field of program verification 
fundamentally miss the point. In a previous chapter, I discussed the misplaced use 
of computer systems as patch solutions to s deep human difficulties. In the similar 
vein, verification proofs deal with only tije. superficial symptoms of system 
Incomprehensible -program "mistakes." 

We have every right to demand the highest standard of reliability from sys- 
terns tJjat we deal, with, and to rea^ire Justification for the use of a particular sys- 
tern; program verification plays a necessary and useful role in satisfying these 

'•"■■■'■ ■■■ ^ •-_•■- .-;•--<: »-i •■■,.-;■-■: ;_,,;■. .,.-.;• .f" «kioi Jftfitioqsfc V«lq lis --r.%: : -.,;. ,j<Ik . ■,. -■•.:• 

demands. But tt Is wise to remember that a "proof" of correctness is not a license 
to establish a computer system as an autonomous decision making entity that Is 
capable of initiating acts (but, not capable of accepting responsibility for them). 
Unless we understand a system and are wffling^to accept responsibility for It, we 
should resist th^ temptations to depend on it. 
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6.2: Psychological Factors of Programming 

My discussion In the preceding sections of this chapter has focused on the 
mechanical component of computer systems; Gerald Weinberg, among others, has 
emphasized nontechnical aspects of systems. The underlying message in his book, 
The Psychology of Computer Programming, Is simple and obvious but nonetheless 
underemphasized: computer programming is fundamentally a human activity, and a 
lack of attention to the psychological aspects of this activity and to the social 
environment In which It takes place can result In the creation of computer programs 
which are undesirable from many viewpoints. 

Early on, Weinberg acknowledges the impossibility of writing "perfect" com- 
puter programs and the need for computer programmers to recognize the limitations 
of their work: "Thus, there are degrees of meeting specifications - of 
'working 1 - and evaluation of programs must take the type of imperfection Into 
account." 86 The acceptable degree of conformance to formal program 
specifications should be made explicit In the course of designing a system. If the 
users of a system are made aware of the ways in which the actual system does 
and does not conform to the proposed specifications of system behavior, then they 
may make more informed decisions about their use of, dependence on, and trust in 

computer output. 

One of the main concepts that Weinberg discusses Is that of "egoless 
programming" - the training of software workers "to accept their humanity - their 
inability to function like a machine - and to value it and work with others so as to 
keep It under the kind of control needed if programming is to be successful 
the problem of the ego must be overcome by a restructuring of the social 



36 Gerald Weinberg, The Psychology of Computer Programming (Mew York: 
Van Nostrand Reinhold Company, 1971), p. 19. 
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environment and, through this means, a restructuring of the value system of the 
programmers in that environment." 86 The goal here Is to debunk stereotypes that 
portray computer programmers as solitary workers whose expertise Is best exer- 
cised in isolation (or, at least, in isolation from other people, though not necessarily 
from computers). Weinberg thinks that programmers should not be encouraged to 

37 

identify themselves too personally with the programs they write, since this can 
discourage cooperation between programmers, particularly in uncovering program 
"bugs" at an early stage of coding. Furthermore, Weinberg believes that egoless 
programming can result in faster average debugging time, more accurate estimates 
of the progress of a programming project as the work proceeds, and the generation 
of more reliable software. Weinberg continues to de-emphasize the role of the soli- 
tary (and potentially indispensable) programmer by stressing team involvement in 
setting goals. In addition, he advocates selecting workers who fit well within a 
shifting environment and are willing to work together. 

It is clear that Weinberg's major point is that effective communication 
between the different members of a programming project and between system 
workers and users Is necessary If we are to strive for better quality computer sys- 
tems. This key issue is given relatively low status in a typical software project. 
For instance, documentation is frequently considered to be among the least impor- 
tant tasks associated with the generation of a new computer system, and certainly 
peripheral to the "real" work of designing and programming the system; instead, 
Weinberg believes that it should be elevated to a professional status of its own. 



86 /6/d., pp. 66-57. 

^This does not mean that programmers should not accept personal respon- 
sibility for their work, but rather that their Identification with their programs should 
be on e prof esstonsJ, and not an overly emotional, level. 
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As is the case with most activities involving human beings, "what is needed hi a 
programming project is slow, careful communication." 38 The social behavior of 
software workers may need to be modified to facilitate more productive communica- 
tion between them. 

6.3: Modern Programming Practices 

Evidence, taken from existing computer applications, about the quality of 
current software Is almost uniformly discouraging. Work in computer system reliabil- 
ity and program verification Is, for the most part, still in the research stage of 
development, and interest In improving programming environments remains primarily 

academic. 

In spite of methodological improvements such as structured 
design and coding, chief programmer teams, online program 
development systems, high level languages and data base 
managers .... the delivered quality of large scale software, 
whether new or modified, remains disgraceful, except where the 
projects or the people involved are specialty chosen. Per- 
vasive cynicism about software Is the Justifiable consequence 
of the many situations where poor results follow long de- 
lays .... Technical panaceas have failed consistency in the 
past and promise to do so for the foreseeable future." 

Since software maintenance currently accounts for more than fifty percent (and 

40 
sometimes as much as eighty percent) of a typical data processing budget, 

there is obviously a strong Incentive for the production of software that works. 

Why, then, is the quality of current computer software so poor? 

The first thing to examine In attempting to answer this question Is the goal 

of software that "works." Historically, this has been translated to mean nothing 



38 Weinberg, p. 1 09. 

39 D. H. McNeil, "Adopting a System Release Discipline," Oatsmat/on, Vol. 26, 
No. 1 (January, 1879), pp. 111-112. 

^ Ibid. ,p. 112. 
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more than software that produces a tolerable approximation of the desired outputs. 
The predominant emphasis on minimizing the cost of computer system generation 
(for instance, incentive pay for early completion) has encouraged the satisfaction 
of the "workability" criterion through means of questionable reliability. Over- 

-*' .,-..*'* - . ■ _.. - - ■ 

designing a first installation of a new system (building in - perhaps through some 
redundancy in the system - higher levels of flexibility, reliability, safety, etc. than 
may necessarily be needed, to handle unanticipated difficulties) is not immediately 
cost effective. Issues like dependability and comprehensibility are long term con- 
cerns not directly related to the market value of a system; because of this, com- 
puter manufacturers have often opted not to devote much of their energies to the 
refinement of failsafe and failure-proof techniques. Flexibility, modifkibility, and 
maintainability have been treated as secondary components of quality assurance; 
primary components are those that relate to at system's Immediate performance. 

The emphasis in a computer system project has always been on coding the 
system software. Productivity Indices typfeaHy measure Unas of code written or 
number of compilable modules produced in a given time period. Structured program- 
ming, walk-throughs, and other modern programming practices are encouraging evi- 
dence of the current interest In software engineering, but this is still a fledgling 
discipline. Some programmers feel that the enforcement of these techniques 
succeeds only m rendering the activity of programming more Inflexible toy allowing 
no room for finding and cultivating i one's *>wn programming style. AH too often, ft 
appears that the motivating philosophy behind attempts to upgrade the quality of 
software calls for the implementation of techniques with low overhead and highly 
visible results, without much consideration for their long term value. 

One difficulty Is the single minded emphasis ©it improving 4h# quality of com- 
puter programs, not systems as a whole. The achievement of better software must 
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involve changes In the operation of computer baaed organizations (software 
development and maintenance Is a management Issue, not a purely technical one) 
and In individual habits (/.e., the development of a software engineering attitude 
and not merely the use of appropriate coding techniques). The continual change 
that is the way of life for most of the computer Industry can be handled In more 
reliable ways. Software workers can avoid Incremental patchwork on systems In 
production between scheduled releases of the system, managers can resist the 
pressures to have little improvements pasted onto the current release without 
going through a complete testing cycle, and users can recognize that they cannot 
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demand new system features without paying for them with time and money. The 
common themes in most current discussions of "the software problem" are the need 
for improved education, planning, and communication, involving managers, data pro- 
cessing analysts, programmers, operators, and users. Ultimately, producing better 
computer systems Is a human problem. "In the long run the evolution of quality 
software depends upon people, not on systems analysis techniques, programming 
languages, or operating systems environments." 

6.4: Criticism 

The most intriguing questions about incomprehensibility relate not merely to 
computer programs, but to computer systems and the social environments in which 
they exist. 

One can, with the same technology, design totally different out- 
comes by designing different social support systems .... The 
technology is the same, yet the pattern of use is highly dissimi- 
lar. The crucial decisions are sociological, not technotogteal 



41 «>/d., pp. 112, 114. 
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.... the . . . social questions frequently Involve a difference If 
not a clash of values . . . . It is not a matter of right versus 
wrong but of right versus right. This is what makesmoral deci- 
sions so dlfTtault. And these are the hard questions.* 3 " 

We have seen that there are personal and sbclaf effects of the use of computer 
Systems, as welt as technical effects. Lewis Mttmford suggests reckoning up "the 
human disadvantages and costs, to say nothing of tfte dangers, of ow unqualified 
acceptance of the system itself." 

Autonomous technology offers what It "does "on "'one* condition: that one must 
not . . . ask for [anything] that the system does not offer . . . Once one opts for 
the system, no further choice remakis."* 6 li <*(der to regain control over computer 
technique, we must ccmtimjalry criticize *the fe^m 

tion"; 48 we must consciously decide whether 'or^lWrt tiW is what we want. The 
development of technology may be modiffed only # n we%ldeh bur sphere of moral 
choices by considering other alternatives. This- may be done in many ways; for 
instsnce, by utilizing the computer as only a partial solution to certain problems 
(". . . the computer too could be applied iwf sWiply wherever tite opportunity 
arises, but only where it is deemed in the best interests of socle^*^ 7 ) or by turn- 
ing tiie positive aspects of computer applications to purposes other than the per- 
petuation of the technological system. 



^Daniel Bed* "Hard Questions and Soft Wimls: A Reply to Weizenbaum," 
Chapter 21 in The Future Study on the Impact of Computers and Information Pro- 
cessing, ed. by Michael L. Dertouzos (Cambridge, MA: The MIT Press, in press). 

4 ^umford., "Authoritarian and Democratic Tecjhnlc§," p. 57. 



^IbM. 



46 Joseph Weizenbaum,. "On the Impact of the Computer on Society," Sci- 
ence, 176 (May 12, 1972), p. 812. 

47 Abbe Mowshowltz, 7ne Conquest of With Information Processing In Human 
Affairs (USA: Addison-Wesley Publishing Company, 1976]}, p. 82. 
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The very leisure that the machine now gives ... can be 
profitably used, not for further commitment to still other kinds of 
machine, furnishing automatic recreation, but by doing significant 
forms of work* unprofitable or technically impossible under mass 
production: work dependant upon special skin, knowledge, 
aesthetic sense. 

Even in a technological world, it is possible to criticize the system, but we forfeit 
our right to criticize if we allow technology to dictate its own course. 

6.5: Responsibility 

The necessity for a strong sense of individual moral responsibility 4n today's 

society cannot be overemphasized. One starting point for computer professionals 

might be the set of "Guidelines for Professional Conduct In Information Processing" 

set forth by the Association for Computing Machinery; in the preamble, the ACM 

urges the following: 

The professional person, to uphold and advance the honor, dig- 
nity and effectiveness of the profession in the arts and sch- 
ences of information processing, and U keeping with high stan- 
dards of competence and ethical conduct: WW be honest, 
forthright end impartial; wiH serve wJf&Joyalty his employer, 
clients and the public; will strive to increase the competence 
and prestige of the profession; wHI use his special knowledge 
and skiH for the advancement of human welfare. [my 
emphasis] ■ .-. 

Responsibility Is here defined not merely in a narrow professional sense, but rather 
In a broad sense that takes Into eoepunt the rehitipnshipj||^tweea information pro- 
cessing professionals and employers, clients, otiier professionals, and the public- 
What requires additional thought is the weaning of responsibility in terms of possi- 
ble repercussions on, for instance, programmers. Questions as to whether or not 



^Mumford, "Authoritarian and Democratic Technics," p. 66. 

49 Reprinted fti C. C. Gottlieb and A. Borodin, Social Issues In Computing 
(New York: Academic Press, 1973), pp. 236-237. 
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computer professionals may be sued or put In jail (for example) for the production 
of "unacceptable" computer systems ought to be considered. 

The most Important step in regaining control over te chn olo gy fertile recogni- 
tion that we alone bear responsibility for our technotogioal systems, and we are 
bound - in technological endeavors no less than n other endeavors - by the social 
ethics governing human behavior In society. The employment of computer system* 
in tasks involving people involves a social decision. The field of computer science 
enjoys substantial public support and has significant impacts on, society* It has a 
social responsibility, pyioroover, thhs soctal rssponSIWIrty cannot be de<i*ed - it is 
exercised by default, through tacit approval «f new teehnotogJcai systems, even If 
explicit judgments are not made. 

... the underlying goal of science and technology Is to 
improve 'the quality of life.' This implies that those closest to 
the technology have a special obligation to question the uses 
and consequences of their work, to exert as much beneficial 
Irrnuencs as posatete, to dkect tecrtf KJlce l car devstepm e rrt and 
Its application, and ■wm'^'m%a» f Wmom^mw^t0S^^kmm^ 
not socially constructive. 60 

In questioning our technology, we must critically examine *he particular role 

we play in the system. ". . . only the l^der wrw la r*«^ to a«ep down has s reel 

chance of success." 61 Technological inevitability need not be accepted; mdh/idue) 

decision making Is possible, and const lttrtSS a f^cetfS iry step towards a 

a human-centered tochnotegy. "It to possible, glv*rtcouN^a aitd Irmlgl^ for. 

deny technology the prerogative to foomrtat* man's o u ft B ttofts . -Ursar 

human questfons and f'tfr find naniane ans«*fs^^ ' ^ 



^Gottlieb and Borodin, pp. 241-242. 

61 Weinberg, p. 85. 

62 Jooeph Welzenbaum, "On the Impact of the Computer on Society," Scf- 
eoce, 176 (May 12, 1072), p. 614. 
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6.8: Humanism 

It should be clear by now that the Improvement of computer programming 
techniques is a significant area of current computer science research (and, to a 
much lesser extent, of present computer science practice), but that the emphasis 
of this chapter has moved to the Improvement of humanizing 
"techniques" - actually, attitudes - to apply to the creation and use of computer 

systems. 

In current practice, humanizing a computer system is often done cosmeti- 
cally; for Instance, by having the system communicate with users in a soothing, 
English-like dialect. While this is helpful, meaningful humanlzatlon of computer sys- 
tems must go beyond superficialities; humanizing a system must involve Injecting 
the human element into a technological system. 

... we had better map out a more positive course: namely, the 
reconstltutlon of both our science and our technics in such a 
fashion as to insert the rejected parts of the human personality 
at every stage in the process. This means gladly sacrificing 
mere quantity In order to restore qualitative choice, shifting the 
seat of authority from the mechanical collective to the human 
personality and the autonomous group, favoring variety and eco- 
loglcnl complexity instead of stressing undue ; yn^ormlty and 
standardization, above all, reducing the Insensate drive to 
extend the system itself, Instead of containing 1|, within definite 
human limits . ... We must ask, not what is goodggn- science 
or technology, . , . , but what Is good for roan . ..." 

At the very least, humanlzatlon requires a recognition of human values. Com- 
puters may be used to explore alternative courses of action, but values must be 
included In decision making about the use of computerized "answers." "What Is evh 
dently wanted is a set of balance sheets in which the relative merits of each solu- 
tion to a technological problem are analyzed both on technological grounds such as 

^Mumford, "Authoritarian and Democratic Technics," p. 68. 
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safety, ease of operation, complexity, and esthetics, and on ethical grounds such 
as moral considerations, effects on the quality of human life, liberty and dignity, and 
other human values." 64 We must come to believe that the dictates of human 
beings supersede those of technological systems; there is relatively little compel- 
ling evidence that modern society does believe this. In addition, a strong sense of 
control over the use of technology should be nurtured; if necessary, this should 
involve "cut[ttng] the whole system back to a point at which it will permit human 
alternatives, human interventions, and human destinations for entirely different pur- 
poses from those of the system itself." 

In an organizational context, care must be taken to avoid undesirable effects 
of the increased rigidity, formality, and alienation that frequently accompany com- 
puterization. One should not ignore the nonrational aspects of social conduct by 
administering a social organization according to purely technological criteria. The 
modern bureaucracy is largely irresponsible In Its pursuit of efficiency - indifferent 
to human heeds and unsupporttve of the pr om otion of interaction awtf communication 
across hierarchical levels. Mowshowitz points out the drMcufty of challenging the 
status quo: ". . . one critical feature ts a mechanism fdtr establishing a continuing 
dialogue between managers and workers or ordinary cftfeens, This Is not likely to 
emerge spontaneously, since it presupposes a fundamental shift In values from pro- 
ductivity and efficiency to human welt-belng.* 6 * Two' things are especially important 
In what Mowshowitz and others have said about improving conditions in a technolog- 
ical society. First, communication Is seen as a key factor fti Changing the existing 
situation; one aspect of this is that more W the people who Interact with computer 



^Mowshowitz, pp. 271-272. 

®^umford, "lujtiiorltarlan and Democratic Techntes,**p. S8: 

66 Mowshowitz, p. 201. 
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system should have access to Information about the system, as we* ee the 
to use this Information. Second, perhaps the ultimate deterrent to comprehending 
computer systems is the human value system that places such a premium on tech- 
nology. 

Finally, and most importantly, technological system IncomprehenelbHIty Is a 
human problem, and the most significant and difficult questions that it raises should 
be answerable to humanistic concerns. Any meaningful comprehension of our 
interactions with computer systems must be preceded by a better understanding of 
our own role in a technological society, and this in turn requires an understanding 
of interpersonal difficulties, human priorities, and ethical values. 

What should this teach us, particularly with respect to the 
question of at least preserving if not enhancing human choice In 
human affairs? 

Certainly that the construction of reliable computer 
software awaits, not so much results of research In computer 
science, but ratter a deeper theoretical understanding of the 
human condition. 

Before computer systems can be made truly comprehensible, human systems must 
be better understood. Before we can control our technological systems, we must 
learn to value people more than technics. 



67 Joeeph Weizenbaum, "Human Choice In the Intersttcee of the 
Mogemechtne," p. 14. Lecture presented at the IFIPS Conference on "Human 
Choice and Computers," Vienna, Austria, in June, 1 979. 
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and is now an assistant professor of computer science at Duke University, 
North Carolina. She is also employed as a senior software engineer at 
SofTech, Mass achusetts, and as a vl^rjg''.e^n^t\^vriPA^>ri1 nr 9 ,n,a ' 
Research- program verification, Inciuding testing, the theory and practice of 
proving correctness of programs, and programming methodologies which 
reduce the difficulties of verification. 

C. C. Gottlieb: Gottlieb is a faculty member in the department of computer science 
at the University of Toronto. 

Jeanne Erard Guflahorn: psychologist Gullahorn is presently a f acuity member at 
Michigan State University; she also acts as a consultant for System 
Development Corp., California. She has published several articles, Including 
"Computer simulation of rote conflict resolution." Research - computer simu- 
iation of social behavior, psychok>gy of women, and cross-cultural research. 

John Taylor Guttahorn: sociologist. GuBahorn Is currently a faculty member In sociol- 
ogy and computer instruction at Michigan State University; he also works as 
a consultant for the U.S. State Department and numerous others. His 
research interests Include the use of computers to test and develop 
theories of aocieJ behavior. 
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Arte Nicholas Habermann: computer scientist. Habermann has been a faculty 
member in the Netherlands and at a variety of European universities, and 
now teaches at Carnegie-Mellon University, In addition to working as a con- 
sultant for Westinghouse Corporation and DEC- Research - programming 
languages, programming systems, operating systems. 

John Kemeny: college president. Kemeny worked on the Los Alamos Project and at 
the Institute for Advanced Study (for Einstein). He has taught at the InstH 
tute for Advanced Study and at Dartmouth; presehlfy, he Is president of 
Dartmouth University. Kemeny is a member of numerous professional associ- 
ations and committees (Including HEW and NRC committees), and author of 
numerous books about mathematics, programming, and science. His research 
Interests are In mathematical models for the sc^claf sciences and the philoso- 
phy of science. 

Philip Kraft: computer scientist. Kraft currently teaches sociology at SUNY in 
Binghampton, New York. His research deals wfth class relations of societies 
in general and the way they are played out In the development of technol- 
ogy. His present work Is with representative of trade unions In the area of 
political implications of the widespread Introduction of microprocessor tech- 
nology. Another of Kraft's current Interests is In the role of women and 
minority men in the field of computing. 

Butler Wright Lampson: computer scientist. Lampson has served on the faculty of 
the University of California at Berkeley, and now works as a research fellow 
in computer science at Xerox Palo AJto Research Center, In addition to 
directing system development at Berkeley Computer Corp. 
Research - operating systems and programming languages. 

John H. Lehman: Lehman's background is in computer software development, and he 
has worked with computer systems In a variety of roles, from programmer to 
project manager. He held a variety of data processing, positions while in the 
Air Force; at different times, he has also held staff p<ksWons In planning, pol- 
icy formation, and requirements determination. He Is presently a computer 
consultant. 

Richard J. Upton: Upton has been a faculty member In computer science at Stan- 
ford University and Yale University; presently, he teaches at Berkeley 
University. His research is in data structures^ algorithms, and computational 
complexity. 

Barbara Liskov: computer scientist. Liskov has worked at the MITRE Corp. and In 
Carnegie-Mellon University's department of computer science. She Is 
currently a computer science faculty member at MIT, where she heads the 
computation structures research group. Her research interests are the 
design of programming languages and software development. 

Zohar Manna: computer scientist. Manna has been associated with the Stanford 
Research institute and Stanford University's department of computer sci- 
ence. His research is in the areas of prOg>aW corre^ltoss, program sche- 
mes, algorithms, and mathematical logic. 
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Bruce Mulish; educator, historian. Mazlish, a PhD graduate of Columbia University, 
has been an instructor at the University of Maine, Columbia University, and 
MIT, and has been a faculty member and chairman of the humanities depart- 
ment at MIT. He has served on panels for the National Science Foundation 
and the American Academy of Arts and Sciences. He is the author of 
numerous books; most recently (101%), The Revofuttonary Ascetic- Kissinger, 
The European Mind in American Policty. 

D. H. McNeil: computer scientist. McNeil is now an independent consultant in the 
Philadelphia area. He has worked as a computer programmer, system 
analyst and designer, Instructor for IBM's general systems division, and 
technical advisor for Shared Medical Systems (Where he developed many of 
the concepts of a system release discipline). 

Marvin Minsky: mathematician. Minsky holds academic degrees from the Phillips 
Andover. Academy, Harvard, and Princeton (PhD). He is one of the founders 
of MIT's artificial Intelligence laboratory and served as director of that tab; 
he has also served on the faculty of the mathematics and computer science 
departments at MIT (where he fs currently Donner Professor of Science). 
Author of several books, Minsky was the recipient of the 1970 Turing Award. 
His research is In the areas of artificial inteiflgence, theory of computation, 
psychology, and engineering. 

Abbe Mowshowitz: Mowshowitz is an associate professor of computer science at 
the University of British Columbia and a visiting research associate in the 
department of computer science at Cornell University. He has special 
interests In combinatorial mathematics and graph theory, fh addition to the 
social impact of technology. -~ 

Lewis Mumford: author. Mumford has served on the faculties of Stanford Univer- 
sity. Universfty of Pennsylvania., MIT, and the University of CaHfornia at 
Berkeley, and was a senior fellow at Wesleyan University's Center for 
Advanced Studies. He has been the recipient of numerous award; among 
them are the National Book Award (196£), the US Medal of Freedom (1864), 
the Emerson Thoreau Medal of ttief American Academy of Arts and Sciences 
(1966), the Gold Medal for Belles Lettres of the National Institute of Arts 
and Letters C197QX and the National Medal for Literature (1972). He has 
been a Fellow with numerous professional organizations, is an honorary 
member of t*hl Beta Rapp% and te a decorated Knight of the British Empire 
(honorary). He Is the author of numerous books, tncfutfing The Myth and the 
Machine (1967, 1970 - 2 volumes) and Technics and Civilization (1934). 

AJen Newell: educator. Newell holds academic degrees in physics and industrial 
administration. He has served on the faculty of Carnegie-Melton University 
and was a consultant with the Rand Corp. and with the Xerox Corp. He 
belongs to a variety of professional organization, Has received several pro- 
fessional awards (including the 1976 Turing Award, which he shared with 
Herbert Simon), and is the author of a number of books (including Human 
Problem Solving, written with H. Simon). Research- artificial Ifrtettfgence, 
psychology, programming systems, and computer structures. 
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Robert L Patrick: Patrick is presently a freelance computer specialist based In 
southern California. His clients include companies with products In 
aerospace, computing, finance, and manufacturing, and he has performed 28 
audits of computer centers. He is the author of the 1974 AFIPS Security 
Manual, as well as many books and reports, Including the 1978 NBS study on 
data integrity practices. 

Edwin W. Paxson: Paxson currently works for the Rand Corp. in California. 

Man J. Perils: educator, computer scientist. Perils has academic degrees from 
Carnegie-Mellon Institute of Tectorrotogy, California Institute of Technology, 
and MIT (PhD In mathematics); he now serves as chairman of the computer 
science department at Yale University. Perils Is a past president of the 
Association of Computing, Machinery (ACM), past editor-in-chief of Communica- 
tions of the ACM, member of numerous professional organizations. He has 
served on numerous committees, including those for |be National Institute of 
Health and the National Science Foundation. ^ 

Theodore Roszak: author, editor, educator. Roszak Is a former Guggenheim Fellow 
(1971-72) and is the author of numerous books, Including Sources: An 
Anthology of Contemporary Materials Useful (for Preserving Sanity While 
Braving the Great Technological Wilderness (1972). 

Thomas Brown Sheridan: Sheridan is currently a professor of mechanical engineering 
at MIT; in addition, he acts as a consultant to various coporatlons. His 
research interests include group decision technology, mathematical models of 
the human operator In control systems, and mah-mactiftie interatlons. 

Herbert Alexander Simon: social scientist. Simon has served on the faculty and 
administration of the University of California, the Illinois Institute of Technol- 
ogy, and Carnegle-MeWon University (where he is now a trustee). His 
numerous awards Include the 1976 Turing Award (shared with A. Newell) and 
the 1978 Nobel Prize In Economics, He is a member f numerous profes- 
sional organizations and author of several books, Including The Sciences of 
the Artificial (1988). 

T. D. Sterling: Sterling is currently associated with the department of computer sci- 
ence at Simon Fraser University, British Columbia. 

Howard K. Thompson, Jr.: computer scientist, bJomathematlclan. Thompson earned 
an MD from Columbia University; presently, he Is a professor of medicine at 
Baylor College medical school. Research - computers In medicine. 

Alan Turing: Turing was a British mathematician who developed the Idea of an 
abstract computing machine that can carry out a certain class of computing 
tasks. This machine, known as a Turjng machine, ts now fundamental to the 
study of theoretical computer science. Turing was. involved frr significant 
cryptography work during World War II. Turing published a number of arti- 
cles on intejagent machinery and man a* 
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Wegbrett: Wegbrett Is associated with the Xerox Palo Alto Research Center. 

Richard Waldinger: computer scientist. Waldinger has pursued research in 
mathematics and artificial Intefflgence; presently, he works at the Stanford 
Reeearch Institute. Research - artificial hrrtefflgence, automatic program syn- 
thesis and verification, mechanical theorem proving, and robotics. 

Gerald M. Weinberg: computer scientist Weinberg is professor of computer sys- 
tems at the School of Advanced Technology, State tJtfversfty of New York. 
He has been Involved with various aspects of computer technology since 
1956, including software Impiementatfen and both hardware and software 
design. Dr. Weinberg has written a number of books on computer program- 
ming, phis several dozen articles on computing, problem solving and systems 
theory. A member of ACfii and AAAS, he tsalsO a *, as weB as former 

Secretary, of the Society for Genera! Syst&W Research. 

Joseph Webwnbaum: Wekenbaum received Ms h^Mr etjucatk* at Wayne State 
University. He has worked oh the development ^ of *a"Var!ety of computer 
systems with Wayne State University, Computer Control Co., Bendix Com- 
puter Division, and the 6E Computer Department (Where he helpetf develop 
programming systems to aid In the design of the computer system which was 
to be built for the Bank df America). He has he# academic appointments 
with Stanford University, Harvard University, the fecrWcMl Institute of Berlin, 
and MIT (where he is presently a professor of computer science). He Is a 
member of numerous professional and pdBfJcal orfla i tf /a tto na, af»d author of 
many articles as wefl aa the book, Cottifkitet Powsr and Human tbastmt From 
Judgment to Calculation (1972). Once known maWy for computer science 
applications such as ELIZA ta ha&ifar language proce^sfcg system), 
Welzenbaum's Interests have shifted In recent years to the area of societal 
Implications of c in particular, i&Pf&a ws«es of responsibility, 

Incomprehensftjfflty, techncfogtcar Inevitability, and the implications of trusting 
computer systems. ;'" f' J __ 

Harvey Wheeler, Jr.: political scientist. Wheeler has taught at John Hopkins Univer- 
sity and at Washington and Lee. He Is currently a senior f allow with the 
Center for the Study of Democratic Institutions In Caflforrrfa and a freelance 
consultant His research Is In the area of potttlcal theory. 

Norbert Wiener: Wiener studied logic with Bertrand Russell and was on the faculty 
of MITa mathematics department. He demonstrated an early interest in 
analogies between electronic and bio! t?es, Wiener's belief ^at 

there was an essential unity of projMtMsj centering around communication 
and control f- in machine or Bvlng tissue - ted to hfe foundation of the scl- 
ence of cybernetics. 

Winner: Winner Is a writer, teacher, and sometimes music critic. He has 
worked for Rosing Stone magazine, the Pentagon, and the M.I.T. He is the 
author of Avtono rectoo/ogy; Naif'^jbntrof as a theme In po/- 

Itlcmt thouont and numerous articles t> Strd the human condition. 

Mr. Winner ia currentty wrfting ' : i «iik 3 on technological design atid the quality 
of pubttc Rfe. 
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Lawrence Yelowltz: Yelowltz holds academic degrees in mathematics (BS and MAT) 
and computer science (PhD). He has been involved with industrial and 
governmental computer programming applications, as weM as research work 
with IBM and the Al group at the National Institute of Health. He has taught 
computer science at New Mexico Institute of Mining and Technology and at 
the University of California at Irvine, and is now an assistant professor of 
computer science at the University of Pittsburgh. A member of several pro- 
fessional organizations, Yelowitz was the first recipient of the Samuel N. 
Alexander Memorial Award sponsored by the Washington, D.C. chapter of 
ACM. Yelowitz's publications are in the area of program correctness. 
Research - program correctness, operating systems, and data structures. 



The information in this section was compiled from the card catalog of MIT's com- 
puter science reading room, and from the following sources: 

Who's Who in America, 40th edition (1978-79). (Illinois: Marquis Who's Who, Inc.). 

American Men and Women of Science, 13th edition. Jaques Cattell Press, ed. 
(New York: R. R. Bowker Company, 1976). 

American Men and Women of Science, 1 2th edition, The Social and Biological Sci- 
ences. Jaques Cattell Press, ed. (New York: R. R. Bowker Company, 1973). 

American Men and Women of Science, 12th edition, Physial and Biological Sciences. 
Jaques Cattell Press, ed. (New York: R. R. Bowker Company, 1972). 
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